2024-04-25T06:53:00.696 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T06:53:00.697 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} --archive /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672923 --owner scheduled_nmordech@teuthology --name nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi --verbose -- /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672923/orig.config.yaml 2024-04-25T06:53:00.720 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T06:53:00.798 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672923 branch: main description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: null first_in_suite: false job_id: '7672923' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f workunit: branch: wip-nitzan-osd-recovery-standalone-test-wait-for-too-full sha1: a08304379f921e5e229e92c75ede8800059313ff owner: scheduled_nmordech@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 7777 sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f sleep_before_teardown: 0 suite: rados:standalone suite_branch: wip-nitzan-osd-recovery-standalone-test-wait-for-too-full suite_path: /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa suite_relpath: qa suite_repo: https://github.com/NitzanMordhai/ceph.git suite_sha1: a08304379f921e5e229e92c75ede8800059313ff targets: smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMthPnGYvuMk54xjeF/AZCFrHdhGdmA4Fk1k7V6BcpUSKk+Ujs9J8mbmHl/xkt2aml0gYg20Yu7aWH89rDSX85k= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_06:28:28 tube: smithi user: nmordech verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T06:53:00.798 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa; will attempt to use it 2024-04-25T06:53:00.799 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa/tasks 2024-04-25T06:53:00.799 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T06:53:00.801 INFO:teuthology.task.internal:Checking packages... 2024-04-25T06:53:00.823 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f' 2024-04-25T06:53:00.823 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T06:53:00.824 INFO:teuthology.packaging:ref: None 2024-04-25T06:53:00.824 INFO:teuthology.packaging:tag: None 2024-04-25T06:53:00.824 INFO:teuthology.packaging:branch: main 2024-04-25T06:53:00.824 INFO:teuthology.packaging:sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T06:53:00.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-04-25T06:53:01.163 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3253-g4e213a58-1jammy 2024-04-25T06:53:01.165 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T06:53:01.174 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T06:53:01.174 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T06:53:01.242 INFO:teuthology.task.internal:Saving configuration 2024-04-25T06:53:01.252 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T06:53:01.276 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T06:53:01.302 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672923', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 06:47:25.977265', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO2yfqM45QagfOIdCIgm1dPv+ho5wujutRwoWrkLSJwb'} 2024-04-25T06:53:01.302 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T06:53:01.313 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-25T06:53:01.313 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T06:53:01.376 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc663e433a0>, signals=[15]) 2024-04-25T06:53:01.376 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T06:53:01.387 INFO:teuthology.task.internal:Opening connections... 2024-04-25T06:53:01.387 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T06:53:01.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:53:01.468 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T06:53:01.480 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-04-25T06:53:01.574 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-04-25T06:53:01.575 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-04-25T06:53:01.623 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T06:53:01.623 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="Ubuntu" 2024-04-25T06:53:01.623 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="22.04" 2024-04-25T06:53:01.623 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T06:53:01.623 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_CODENAME=jammy 2024-04-25T06:53:01.623 INFO:teuthology.orchestra.run.smithi102.stdout:ID=ubuntu 2024-04-25T06:53:01.623 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE=debian 2024-04-25T06:53:01.623 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T06:53:01.623 INFO:teuthology.orchestra.run.smithi102.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T06:53:01.623 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T06:53:01.624 INFO:teuthology.orchestra.run.smithi102.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T06:53:01.624 INFO:teuthology.orchestra.run.smithi102.stdout:UBUNTU_CODENAME=jammy 2024-04-25T06:53:01.624 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-04-25T06:53:01.653 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T06:53:01.664 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T06:53:01.698 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T06:53:01.699 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-04-25T06:53:01.703 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T06:53:01.726 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T06:53:01.726 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-04-25T06:53:01.753 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T06:53:01.784 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T06:53:01.801 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-25T06:53:01.802 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-25T06:53:01.802 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T06:53:01.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T06:53:01.807 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-04-25T06:53:01.855 INFO:teuthology.orchestra.run.smithi102.stdout:5.15.0-56-generic 2024-04-25T06:53:01.855 INFO:teuthology.task.kernel:Running kernel on smithi102: 5.15.0-56-generic 2024-04-25T06:53:01.855 DEBUG:teuthology.orchestra.run.smithi102:> sudo apt-get clean 2024-04-25T06:53:01.991 DEBUG:teuthology.orchestra.run.smithi102:> sudo apt-get update 2024-04-25T06:53:02.285 INFO:teuthology.orchestra.run.smithi102.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T06:53:02.285 INFO:teuthology.orchestra.run.smithi102.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T06:53:02.344 INFO:teuthology.orchestra.run.smithi102.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T06:53:02.701 INFO:teuthology.orchestra.run.smithi102.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T06:53:02.713 INFO:teuthology.orchestra.run.smithi102.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T06:53:02.854 INFO:teuthology.orchestra.run.smithi102.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T06:53:03.125 INFO:teuthology.orchestra.run.smithi102.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T06:53:03.142 INFO:teuthology.orchestra.run.smithi102.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T06:53:03.151 INFO:teuthology.orchestra.run.smithi102.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T06:53:03.152 INFO:teuthology.orchestra.run.smithi102.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T06:53:03.153 INFO:teuthology.orchestra.run.smithi102.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T06:53:03.183 INFO:teuthology.orchestra.run.smithi102.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [619 kB] 2024-04-25T06:53:03.225 INFO:teuthology.orchestra.run.smithi102.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T06:53:03.242 INFO:teuthology.orchestra.run.smithi102.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T06:53:03.246 INFO:teuthology.orchestra.run.smithi102.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T06:53:03.249 INFO:teuthology.orchestra.run.smithi102.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T06:53:03.278 INFO:teuthology.orchestra.run.smithi102.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T06:53:03.279 INFO:teuthology.orchestra.run.smithi102.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T06:53:03.306 INFO:teuthology.orchestra.run.smithi102.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T06:53:03.306 INFO:teuthology.orchestra.run.smithi102.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T06:53:03.307 INFO:teuthology.orchestra.run.smithi102.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T06:53:03.315 INFO:teuthology.orchestra.run.smithi102.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T06:53:03.481 INFO:teuthology.orchestra.run.smithi102.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T06:53:03.488 INFO:teuthology.orchestra.run.smithi102.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T06:53:03.896 INFO:teuthology.orchestra.run.smithi102.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T06:53:03.901 INFO:teuthology.orchestra.run.smithi102.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T06:53:03.902 INFO:teuthology.orchestra.run.smithi102.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T06:53:03.915 INFO:teuthology.orchestra.run.smithi102.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T06:53:03.918 INFO:teuthology.orchestra.run.smithi102.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T06:53:04.152 INFO:teuthology.orchestra.run.smithi102.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T06:53:04.227 INFO:teuthology.orchestra.run.smithi102.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T06:53:04.253 INFO:teuthology.orchestra.run.smithi102.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T06:53:04.254 INFO:teuthology.orchestra.run.smithi102.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T06:53:04.257 INFO:teuthology.orchestra.run.smithi102.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T06:53:04.258 INFO:teuthology.orchestra.run.smithi102.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T06:53:04.259 INFO:teuthology.orchestra.run.smithi102.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T06:53:04.259 INFO:teuthology.orchestra.run.smithi102.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [90.8 kB] 2024-04-25T06:53:04.275 INFO:teuthology.orchestra.run.smithi102.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [82.9 kB] 2024-04-25T06:53:04.280 INFO:teuthology.orchestra.run.smithi102.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T06:53:04.445 INFO:teuthology.orchestra.run.smithi102.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T06:53:04.528 INFO:teuthology.orchestra.run.smithi102.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [29.8 kB] 2024-04-25T06:53:04.649 INFO:teuthology.orchestra.run.smithi102.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [18.5 kB] 2024-04-25T06:53:04.701 INFO:teuthology.orchestra.run.smithi102.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T06:53:04.733 INFO:teuthology.orchestra.run.smithi102.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T06:53:08.952 INFO:teuthology.orchestra.run.smithi102.stdout:Fetched 13.3 MB in 3s (4,983 kB/s) 2024-04-25T06:53:10.222 INFO:teuthology.orchestra.run.smithi102.stdout:Reading package lists... 2024-04-25T06:53:10.238 DEBUG:teuthology.orchestra.run.smithi102:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T06:53:10.301 INFO:teuthology.orchestra.run.smithi102.stdout:Reading package lists... 2024-04-25T06:53:10.501 INFO:teuthology.orchestra.run.smithi102.stdout:Building dependency tree... 2024-04-25T06:53:10.502 INFO:teuthology.orchestra.run.smithi102.stdout:Reading state information... 2024-04-25T06:53:10.676 INFO:teuthology.orchestra.run.smithi102.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T06:53:10.676 INFO:teuthology.orchestra.run.smithi102.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T06:53:10.676 INFO:teuthology.orchestra.run.smithi102.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T06:53:10.677 INFO:teuthology.orchestra.run.smithi102.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T06:53:10.677 INFO:teuthology.orchestra.run.smithi102.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T06:53:10.677 INFO:teuthology.orchestra.run.smithi102.stdout: python2.7-minimal 2024-04-25T06:53:10.677 INFO:teuthology.orchestra.run.smithi102.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T06:53:10.678 INFO:teuthology.orchestra.run.smithi102.stdout:The following additional packages will be installed: 2024-04-25T06:53:10.678 INFO:teuthology.orchestra.run.smithi102.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T06:53:10.678 INFO:teuthology.orchestra.run.smithi102.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T06:53:10.678 INFO:teuthology.orchestra.run.smithi102.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T06:53:10.679 INFO:teuthology.orchestra.run.smithi102.stdout:Suggested packages: 2024-04-25T06:53:10.680 INFO:teuthology.orchestra.run.smithi102.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T06:53:10.680 INFO:teuthology.orchestra.run.smithi102.stdout:Recommended packages: 2024-04-25T06:53:10.680 INFO:teuthology.orchestra.run.smithi102.stdout: thermald 2024-04-25T06:53:10.732 INFO:teuthology.orchestra.run.smithi102.stdout:The following NEW packages will be installed: 2024-04-25T06:53:10.733 INFO:teuthology.orchestra.run.smithi102.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T06:53:10.733 INFO:teuthology.orchestra.run.smithi102.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T06:53:10.733 INFO:teuthology.orchestra.run.smithi102.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T06:53:10.734 INFO:teuthology.orchestra.run.smithi102.stdout:The following packages will be upgraded: 2024-04-25T06:53:10.735 INFO:teuthology.orchestra.run.smithi102.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T06:53:10.842 INFO:teuthology.orchestra.run.smithi102.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T06:53:10.843 INFO:teuthology.orchestra.run.smithi102.stdout:Need to get 113 MB of archives. 2024-04-25T06:53:10.843 INFO:teuthology.orchestra.run.smithi102.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T06:53:10.843 INFO:teuthology.orchestra.run.smithi102.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T06:53:11.747 INFO:teuthology.orchestra.run.smithi102.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T06:53:11.958 INFO:teuthology.orchestra.run.smithi102.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T06:53:13.248 INFO:teuthology.orchestra.run.smithi102.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T06:53:13.248 INFO:teuthology.orchestra.run.smithi102.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T06:53:13.248 INFO:teuthology.orchestra.run.smithi102.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T06:53:13.418 INFO:teuthology.orchestra.run.smithi102.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T06:53:13.535 INFO:teuthology.orchestra.run.smithi102.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T06:53:13.916 INFO:teuthology.orchestra.run.smithi102.stdout:Fetched 113 MB in 3s (40.5 MB/s) 2024-04-25T06:53:14.099 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T06:53:16.192 INFO:teuthology.orchestra.run.smithi102.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T06:53:16.195 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:53:16.313 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:18.448 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T06:53:18.469 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:53:18.553 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:19.026 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T06:53:19.047 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:53:19.104 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:26.095 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:53:26.222 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:53:26.547 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:53:26.667 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:53:26.971 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T06:53:26.993 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T06:53:27.037 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T06:53:32.529 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T06:53:32.566 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:53:32.603 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:35.775 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:53:35.900 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:53:36.233 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T06:53:36.346 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:36.489 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T06:53:36.618 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:38.032 INFO:teuthology.orchestra.run.smithi102.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:53:38.032 INFO:teuthology.orchestra.run.smithi102.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T06:53:38.032 INFO:teuthology.orchestra.run.smithi102.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:53:38.032 INFO:teuthology.orchestra.run.smithi102.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T06:53:38.453 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:39.872 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:41.265 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T06:53:41.366 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T06:53:41.467 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:41.535 INFO:teuthology.orchestra.run.smithi102.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T06:53:41.535 INFO:teuthology.orchestra.run.smithi102.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:53:55.807 INFO:teuthology.orchestra.run.smithi102.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T06:53:55.807 INFO:teuthology.orchestra.run.smithi102.stdout:Sourcing file `/etc/default/grub' 2024-04-25T06:53:55.828 INFO:teuthology.orchestra.run.smithi102.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T06:53:55.850 INFO:teuthology.orchestra.run.smithi102.stdout:Generating grub configuration file ... 2024-04-25T06:53:56.119 INFO:teuthology.orchestra.run.smithi102.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:53:56.143 INFO:teuthology.orchestra.run.smithi102.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:53:56.412 INFO:teuthology.orchestra.run.smithi102.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:53:56.417 INFO:teuthology.orchestra.run.smithi102.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T06:53:56.494 INFO:teuthology.orchestra.run.smithi102.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T06:53:56.500 INFO:teuthology.orchestra.run.smithi102.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T06:53:56.672 INFO:teuthology.orchestra.run.smithi102.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T06:53:56.672 INFO:teuthology.orchestra.run.smithi102.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T06:53:56.672 INFO:teuthology.orchestra.run.smithi102.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T06:53:56.709 INFO:teuthology.orchestra.run.smithi102.stdout:done 2024-04-25T06:53:57.170 DEBUG:teuthology.orchestra.run.smithi102:> dpkg -s linux-image-generic 2024-04-25T06:53:57.197 INFO:teuthology.orchestra.run.smithi102.stdout:Package: linux-image-generic 2024-04-25T06:53:57.197 INFO:teuthology.orchestra.run.smithi102.stdout:Status: install ok installed 2024-04-25T06:53:57.197 INFO:teuthology.orchestra.run.smithi102.stdout:Priority: optional 2024-04-25T06:53:57.197 INFO:teuthology.orchestra.run.smithi102.stdout:Section: kernel 2024-04-25T06:53:57.197 INFO:teuthology.orchestra.run.smithi102.stdout:Installed-Size: 21 2024-04-25T06:53:57.197 INFO:teuthology.orchestra.run.smithi102.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T06:53:57.197 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: amd64 2024-04-25T06:53:57.197 INFO:teuthology.orchestra.run.smithi102.stdout:Source: linux-meta 2024-04-25T06:53:57.197 INFO:teuthology.orchestra.run.smithi102.stdout:Version: 5.15.0.105.102 2024-04-25T06:53:57.198 INFO:teuthology.orchestra.run.smithi102.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T06:53:57.198 INFO:teuthology.orchestra.run.smithi102.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T06:53:57.198 INFO:teuthology.orchestra.run.smithi102.stdout:Recommends: thermald 2024-04-25T06:53:57.198 INFO:teuthology.orchestra.run.smithi102.stdout:Description: Generic Linux kernel image 2024-04-25T06:53:57.198 INFO:teuthology.orchestra.run.smithi102.stdout: This package will always depend on the latest generic kernel image 2024-04-25T06:53:57.198 INFO:teuthology.orchestra.run.smithi102.stdout: available. 2024-04-25T06:53:57.199 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T06:53:57.199 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T06:53:57.199 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T06:53:57.199 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-25T06:53:57.199 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi102.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T06:53:57.199 DEBUG:teuthology.orchestra.run.smithi102:> sudo apt-get clean 2024-04-25T06:53:57.373 DEBUG:teuthology.orchestra.run.smithi102:> sudo apt-get update 2024-04-25T06:53:57.564 INFO:teuthology.orchestra.run.smithi102.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T06:53:57.611 INFO:teuthology.orchestra.run.smithi102.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T06:53:57.611 INFO:teuthology.orchestra.run.smithi102.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T06:53:57.669 INFO:teuthology.orchestra.run.smithi102.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T06:53:58.931 INFO:teuthology.orchestra.run.smithi102.stdout:Reading package lists... 2024-04-25T06:53:58.949 DEBUG:teuthology.orchestra.run.smithi102:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T06:53:59.000 INFO:teuthology.orchestra.run.smithi102.stdout:Reading package lists... 2024-04-25T06:53:59.179 INFO:teuthology.orchestra.run.smithi102.stdout:Building dependency tree... 2024-04-25T06:53:59.179 INFO:teuthology.orchestra.run.smithi102.stdout:Reading state information... 2024-04-25T06:53:59.333 INFO:teuthology.orchestra.run.smithi102.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T06:53:59.333 INFO:teuthology.orchestra.run.smithi102.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T06:53:59.333 INFO:teuthology.orchestra.run.smithi102.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T06:53:59.333 INFO:teuthology.orchestra.run.smithi102.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T06:53:59.334 INFO:teuthology.orchestra.run.smithi102.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T06:53:59.334 INFO:teuthology.orchestra.run.smithi102.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T06:53:59.334 INFO:teuthology.orchestra.run.smithi102.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T06:53:59.334 INFO:teuthology.orchestra.run.smithi102.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T06:53:59.372 INFO:teuthology.orchestra.run.smithi102.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T06:53:59.373 DEBUG:teuthology.orchestra.run.smithi102:> dpkg -s linux-image-generic 2024-04-25T06:53:59.429 INFO:teuthology.orchestra.run.smithi102.stdout:Package: linux-image-generic 2024-04-25T06:53:59.429 INFO:teuthology.orchestra.run.smithi102.stdout:Status: install ok installed 2024-04-25T06:53:59.429 INFO:teuthology.orchestra.run.smithi102.stdout:Priority: optional 2024-04-25T06:53:59.429 INFO:teuthology.orchestra.run.smithi102.stdout:Section: kernel 2024-04-25T06:53:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:Installed-Size: 21 2024-04-25T06:53:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T06:53:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:Architecture: amd64 2024-04-25T06:53:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:Source: linux-meta 2024-04-25T06:53:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:Version: 5.15.0.105.102 2024-04-25T06:53:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T06:53:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T06:53:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:Recommends: thermald 2024-04-25T06:53:59.430 INFO:teuthology.orchestra.run.smithi102.stdout:Description: Generic Linux kernel image 2024-04-25T06:53:59.430 INFO:teuthology.orchestra.run.smithi102.stdout: This package will always depend on the latest generic kernel image 2024-04-25T06:53:59.430 INFO:teuthology.orchestra.run.smithi102.stdout: available. 2024-04-25T06:53:59.431 DEBUG:teuthology.orchestra.run.smithi102:> mktemp 2024-04-25T06:53:59.475 INFO:teuthology.orchestra.run.smithi102.stdout:/tmp/tmp.3ScjzLDY3y 2024-04-25T06:53:59.475 DEBUG:teuthology.orchestra.run.smithi102:> sudo cp /boot/grub/grub.cfg /tmp/tmp.3ScjzLDY3y 2024-04-25T06:53:59.530 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 0666 /tmp/tmp.3ScjzLDY3y 2024-04-25T06:53:59.648 DEBUG:teuthology.orchestra.remote:smithi102:/tmp/tmp.3ScjzLDY3y is 10KB 2024-04-25T06:53:59.660 DEBUG:teuthology.orchestra.run.smithi102:> rm -fr /tmp/tmp.3ScjzLDY3y 2024-04-25T06:53:59.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T06:53:59.717 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T06:53:59.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T06:53:59.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T06:53:59.779 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T06:53:59.780 DEBUG:teuthology.orchestra.run.smithi102:> sudo update-grub 2024-04-25T06:54:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:Sourcing file `/etc/default/grub' 2024-04-25T06:54:01.027 INFO:teuthology.orchestra.run.smithi102.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T06:54:01.039 INFO:teuthology.orchestra.run.smithi102.stderr:Generating grub configuration file ... 2024-04-25T06:54:01.206 INFO:teuthology.orchestra.run.smithi102.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:54:01.211 INFO:teuthology.orchestra.run.smithi102.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:54:01.430 INFO:teuthology.orchestra.run.smithi102.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:54:01.435 INFO:teuthology.orchestra.run.smithi102.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T06:54:01.476 INFO:teuthology.orchestra.run.smithi102.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T06:54:01.481 INFO:teuthology.orchestra.run.smithi102.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T06:54:01.616 INFO:teuthology.orchestra.run.smithi102.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T06:54:01.616 INFO:teuthology.orchestra.run.smithi102.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T06:54:01.616 INFO:teuthology.orchestra.run.smithi102.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T06:54:01.638 INFO:teuthology.orchestra.run.smithi102.stderr:done 2024-04-25T06:54:01.647 DEBUG:teuthology.orchestra.run.smithi102:> sudo shutdown -r now 2024-04-25T06:54:31.679 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-25T06:54:31.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T06:54:31.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:55:31.743 DEBUG:teuthology.orchestra.remote:timed out 2024-04-25T06:55:40.750 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T06:55:40.751 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:55:41.106 ERROR:teuthology.orchestra.connection:Error authenticating with smithi102.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 794, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1664, in auth_publickey raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2024-04-25T06:55:45.185 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-04-25T06:55:46.053 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T06:55:46.053 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-25T06:55:46.053 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2024-04-25T06:55:46.100 INFO:teuthology.orchestra.run.smithi102.stdout:5.15.0-105-generic 2024-04-25T06:55:46.101 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T06:55:46.101 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T06:55:46.101 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T06:55:47.101 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T06:55:47.102 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T06:55:47.174 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2024-04-25T06:55:47.198 DEBUG:teuthology.parallel:result is None 2024-04-25T06:55:47.199 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T06:55:47.207 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T06:55:47.207 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T06:55:47.212 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T06:55:47.227 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T06:55:47.257 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T06:55:47.257 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T06:55:47.277 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T06:55:47.291 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T06:55:47.291 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T06:55:47.347 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:55:47.355 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:55:47.356 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T06:55:47.363 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T06:55:47.364 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T06:55:47.414 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T06:55:47.423 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T06:55:47.424 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T06:55:47.461 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T06:55:47.509 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T06:55:47.557 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T06:55:47.557 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T06:55:47.622 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-04-25T06:55:47.699 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T06:55:47.706 INFO:teuthology.task.internal:Starting timer... 2024-04-25T06:55:47.707 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T06:55:47.754 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T06:55:47.801 DEBUG:teuthology.task.selinux:Excluding smithi102: OS 'ubuntu' does not support SELinux 2024-04-25T06:55:47.802 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T06:55:47.802 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-25T06:55:47.802 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T06:55:47.802 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T06:55:47.835 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-25T06:55:47.841 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-25T06:55:47.967 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T06:55:47.983 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T06:55:47.985 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi102.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T07:02:41.187 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi102.front.sepia.ceph.com')] 2024-04-25T07:02:41.188 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T07:02:41.189 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T07:02:41.269 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-04-25T07:02:41.343 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-04-25T07:02:41.343 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T07:02:41.353 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T07:02:41.354 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T07:02:41.354 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T07:02:41.410 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T07:02:41.410 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: Command line: ntpd -gq 2024-04-25T07:02:41.410 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: ---------------------------------------------------- 2024-04-25T07:02:41.410 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T07:02:41.410 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T07:02:41.410 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: corporation. Support and training for ntp-4 are 2024-04-25T07:02:41.410 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: available at https://www.nwtime.org/support 2024-04-25T07:02:41.411 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: ---------------------------------------------------- 2024-04-25T07:02:41.411 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: proto: precision = 0.057 usec (-24) 2024-04-25T07:02:41.411 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: basedate set to 2022-02-04 2024-04-25T07:02:41.411 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: gps base set to 2022-02-06 (week 2196) 2024-04-25T07:02:41.411 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T07:02:41.411 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T07:02:41.411 INFO:teuthology.orchestra.run.smithi102.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T07:02:41.412 INFO:teuthology.orchestra.run.smithi102.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T07:02:41.412 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T07:02:41.412 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T07:02:41.413 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T07:02:41.413 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: Listen normally on 3 enp3s0f1 172.21.15.102:123 2024-04-25T07:02:41.413 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: Listen normally on 4 lo [::1]:123 2024-04-25T07:02:41.413 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d769%5]:123 2024-04-25T07:02:41.413 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:41 ntpd[17229]: Listening on routing socket on fd #22 for interface updates 2024-04-25T07:02:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:42.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:42.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:42.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:42.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:42 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:44.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:44.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:44.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:44.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:44.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:44.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:44.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:44.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:44.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:44.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:44.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:44.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:44.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:44.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:44 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:46.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:46.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:46.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:46 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:48.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:48.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:48.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:48.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:48.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:48.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:48.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:48.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:48.411 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:48.412 INFO:teuthology.orchestra.run.smithi102.stdout:25 Apr 07:02:48 ntpd[17229]: ntpd: time slew -0.001924 s 2024-04-25T07:02:48.412 INFO:teuthology.orchestra.run.smithi102.stdout:ntpd: time slew -0.001924s 2024-04-25T07:02:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T07:02:48.412 INFO:teuthology.orchestra.run.smithi102.stderr:25 Apr 07:02:48 ntpd[17229]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T07:02:48.466 INFO:teuthology.orchestra.run.smithi102.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T07:02:48.466 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================== 2024-04-25T07:02:48.466 INFO:teuthology.orchestra.run.smithi102.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:02:48.466 INFO:teuthology.orchestra.run.smithi102.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:02:48.466 INFO:teuthology.orchestra.run.smithi102.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:02:48.466 INFO:teuthology.orchestra.run.smithi102.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:02:48.467 INFO:teuthology.run_tasks:Running task install... 2024-04-25T07:02:48.477 DEBUG:teuthology.task.install:project ceph 2024-04-25T07:02:48.478 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f'}} 2024-04-25T07:02:48.478 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f'} 2024-04-25T07:02:48.478 INFO:teuthology.task.install:Using flavor: default 2024-04-25T07:02:48.483 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T07:02:48.483 INFO:teuthology.task.install:extra packages: [] 2024-04-25T07:02:48.484 DEBUG:teuthology.orchestra.run.smithi102:> sudo apt-key list | grep Ceph 2024-04-25T07:02:48.563 INFO:teuthology.orchestra.run.smithi102.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-25T07:02:48.590 INFO:teuthology.orchestra.run.smithi102.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-25T07:02:48.591 INFO:teuthology.orchestra.run.smithi102.stdout:uid [ unknown] Ceph.com (release key) 2024-04-25T07:02:48.591 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-25T07:02:48.591 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T07:02:48.738 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default/ 2024-04-25T07:02:48.738 INFO:teuthology.task.install.deb:Package version is 19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:02:48.971 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T07:02:48.971 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-25T07:02:48.985 DEBUG:teuthology.orchestra.run.smithi102:> sudo apt-get update 2024-04-25T07:02:49.283 INFO:teuthology.orchestra.run.smithi102.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T07:02:49.283 INFO:teuthology.orchestra.run.smithi102.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T07:02:49.290 INFO:teuthology.orchestra.run.smithi102.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy InRelease 2024-04-25T07:02:49.327 INFO:teuthology.orchestra.run.smithi102.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-25T07:02:49.356 INFO:teuthology.orchestra.run.smithi102.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T07:02:49.364 INFO:teuthology.orchestra.run.smithi102.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-25T07:02:49.404 INFO:teuthology.orchestra.run.smithi102.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-04-25T07:02:49.452 INFO:teuthology.orchestra.run.smithi102.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,426 B] 2024-04-25T07:02:49.718 INFO:teuthology.orchestra.run.smithi102.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T07:02:49.871 INFO:teuthology.orchestra.run.smithi102.stdout:Fetched 258 kB in 1s (369 kB/s) 2024-04-25T07:02:50.888 INFO:teuthology.orchestra.run.smithi102.stdout:Reading package lists... 2024-04-25T07:02:50.905 DEBUG:teuthology.orchestra.run.smithi102:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3227-gc37f63b3-1jammy cephadm=19.0.0-3227-gc37f63b3-1jammy ceph-mds=19.0.0-3227-gc37f63b3-1jammy ceph-mgr=19.0.0-3227-gc37f63b3-1jammy ceph-common=19.0.0-3227-gc37f63b3-1jammy ceph-fuse=19.0.0-3227-gc37f63b3-1jammy ceph-test=19.0.0-3227-gc37f63b3-1jammy ceph-volume=19.0.0-3227-gc37f63b3-1jammy radosgw=19.0.0-3227-gc37f63b3-1jammy python3-rados=19.0.0-3227-gc37f63b3-1jammy python3-rgw=19.0.0-3227-gc37f63b3-1jammy python3-cephfs=19.0.0-3227-gc37f63b3-1jammy python3-rbd=19.0.0-3227-gc37f63b3-1jammy libcephfs2=19.0.0-3227-gc37f63b3-1jammy libcephfs-dev=19.0.0-3227-gc37f63b3-1jammy librados2=19.0.0-3227-gc37f63b3-1jammy librbd1=19.0.0-3227-gc37f63b3-1jammy rbd-fuse=19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:02:50.966 INFO:teuthology.orchestra.run.smithi102.stdout:Reading package lists... 2024-04-25T07:02:51.166 INFO:teuthology.orchestra.run.smithi102.stdout:Building dependency tree... 2024-04-25T07:02:51.167 INFO:teuthology.orchestra.run.smithi102.stdout:Reading state information... 2024-04-25T07:02:51.360 INFO:teuthology.orchestra.run.smithi102.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T07:02:51.360 INFO:teuthology.orchestra.run.smithi102.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T07:02:51.360 INFO:teuthology.orchestra.run.smithi102.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T07:02:51.360 INFO:teuthology.orchestra.run.smithi102.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T07:02:51.360 INFO:teuthology.orchestra.run.smithi102.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T07:02:51.361 INFO:teuthology.orchestra.run.smithi102.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T07:02:51.361 INFO:teuthology.orchestra.run.smithi102.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T07:02:51.361 INFO:teuthology.orchestra.run.smithi102.stdout:The following additional packages will be installed: 2024-04-25T07:02:51.361 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-25T07:02:51.361 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-25T07:02:51.362 INFO:teuthology.orchestra.run.smithi102.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T07:02:51.362 INFO:teuthology.orchestra.run.smithi102.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T07:02:51.363 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-25T07:02:51.364 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-25T07:02:51.364 INFO:teuthology.orchestra.run.smithi102.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-25T07:02:51.364 INFO:teuthology.orchestra.run.smithi102.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-25T07:02:51.364 INFO:teuthology.orchestra.run.smithi102.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-25T07:02:51.364 INFO:teuthology.orchestra.run.smithi102.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-25T07:02:51.364 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-25T07:02:51.364 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet zip 2024-04-25T07:02:51.364 INFO:teuthology.orchestra.run.smithi102.stdout:Suggested packages: 2024-04-25T07:02:51.364 INFO:teuthology.orchestra.run.smithi102.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-25T07:02:51.364 INFO:teuthology.orchestra.run.smithi102.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-25T07:02:51.364 INFO:teuthology.orchestra.run.smithi102.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-25T07:02:51.365 INFO:teuthology.orchestra.run.smithi102.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-25T07:02:51.365 INFO:teuthology.orchestra.run.smithi102.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-25T07:02:51.365 INFO:teuthology.orchestra.run.smithi102.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-25T07:02:51.365 INFO:teuthology.orchestra.run.smithi102.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-25T07:02:51.365 INFO:teuthology.orchestra.run.smithi102.stdout:Recommended packages: 2024-04-25T07:02:51.365 INFO:teuthology.orchestra.run.smithi102.stdout: btrfs-tools 2024-04-25T07:02:51.463 INFO:teuthology.orchestra.run.smithi102.stdout:The following NEW packages will be installed: 2024-04-25T07:02:51.463 INFO:teuthology.orchestra.run.smithi102.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-25T07:02:51.463 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-25T07:02:51.463 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-25T07:02:51.463 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-25T07:02:51.463 INFO:teuthology.orchestra.run.smithi102.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-25T07:02:51.463 INFO:teuthology.orchestra.run.smithi102.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-25T07:02:51.464 INFO:teuthology.orchestra.run.smithi102.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-25T07:02:51.464 INFO:teuthology.orchestra.run.smithi102.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-25T07:02:51.464 INFO:teuthology.orchestra.run.smithi102.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-25T07:02:51.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-25T07:02:51.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-25T07:02:51.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T07:02:51.464 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-25T07:02:51.465 INFO:teuthology.orchestra.run.smithi102.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-25T07:02:51.590 INFO:teuthology.orchestra.run.smithi102.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-25T07:02:51.590 INFO:teuthology.orchestra.run.smithi102.stdout:Need to get 181 MB of archives. 2024-04-25T07:02:51.590 INFO:teuthology.orchestra.run.smithi102.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-25T07:02:51.590 INFO:teuthology.orchestra.run.smithi102.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-25T07:02:51.605 INFO:teuthology.orchestra.run.smithi102.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3227-gc37f63b3-1jammy [3,556 kB] 2024-04-25T07:02:51.845 INFO:teuthology.orchestra.run.smithi102.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-25T07:02:51.854 INFO:teuthology.orchestra.run.smithi102.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-25T07:02:51.890 INFO:teuthology.orchestra.run.smithi102.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-25T07:02:52.023 INFO:teuthology.orchestra.run.smithi102.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-25T07:02:52.031 INFO:teuthology.orchestra.run.smithi102.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-25T07:02:52.048 INFO:teuthology.orchestra.run.smithi102.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3227-gc37f63b3-1jammy [3,224 kB] 2024-04-25T07:02:52.051 INFO:teuthology.orchestra.run.smithi102.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-25T07:02:52.057 INFO:teuthology.orchestra.run.smithi102.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-25T07:02:52.058 INFO:teuthology.orchestra.run.smithi102.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-25T07:02:52.060 INFO:teuthology.orchestra.run.smithi102.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-25T07:02:52.129 INFO:teuthology.orchestra.run.smithi102.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-25T07:02:52.146 INFO:teuthology.orchestra.run.smithi102.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-25T07:02:52.150 INFO:teuthology.orchestra.run.smithi102.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-25T07:02:52.152 INFO:teuthology.orchestra.run.smithi102.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-25T07:02:52.153 INFO:teuthology.orchestra.run.smithi102.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-25T07:02:52.157 INFO:teuthology.orchestra.run.smithi102.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-25T07:02:52.173 INFO:teuthology.orchestra.run.smithi102.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-25T07:02:52.177 INFO:teuthology.orchestra.run.smithi102.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-25T07:02:52.178 INFO:teuthology.orchestra.run.smithi102.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-25T07:02:52.178 INFO:teuthology.orchestra.run.smithi102.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-25T07:02:52.219 INFO:teuthology.orchestra.run.smithi102.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-25T07:02:52.219 INFO:teuthology.orchestra.run.smithi102.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-25T07:02:52.220 INFO:teuthology.orchestra.run.smithi102.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-25T07:02:52.221 INFO:teuthology.orchestra.run.smithi102.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-25T07:02:52.221 INFO:teuthology.orchestra.run.smithi102.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-25T07:02:52.222 INFO:teuthology.orchestra.run.smithi102.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-25T07:02:52.224 INFO:teuthology.orchestra.run.smithi102.stdout:Get:29 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3227-gc37f63b3-1jammy [712 kB] 2024-04-25T07:02:52.260 INFO:teuthology.orchestra.run.smithi102.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-25T07:02:52.262 INFO:teuthology.orchestra.run.smithi102.stdout:Get:31 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3227-gc37f63b3-1jammy [356 kB] 2024-04-25T07:02:52.265 INFO:teuthology.orchestra.run.smithi102.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-25T07:02:52.266 INFO:teuthology.orchestra.run.smithi102.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-25T07:02:52.267 INFO:teuthology.orchestra.run.smithi102.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-25T07:02:52.280 INFO:teuthology.orchestra.run.smithi102.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3227-gc37f63b3-1jammy [32.8 kB] 2024-04-25T07:02:52.281 INFO:teuthology.orchestra.run.smithi102.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3227-gc37f63b3-1jammy [180 kB] 2024-04-25T07:02:52.286 INFO:teuthology.orchestra.run.smithi102.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3227-gc37f63b3-1jammy [67.1 kB] 2024-04-25T07:02:52.287 INFO:teuthology.orchestra.run.smithi102.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3227-gc37f63b3-1jammy [330 kB] 2024-04-25T07:02:52.301 INFO:teuthology.orchestra.run.smithi102.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-25T07:02:52.303 INFO:teuthology.orchestra.run.smithi102.stdout:Get:40 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3227-gc37f63b3-1jammy [6,893 kB] 2024-04-25T07:02:52.303 INFO:teuthology.orchestra.run.smithi102.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-25T07:02:52.304 INFO:teuthology.orchestra.run.smithi102.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-25T07:02:52.305 INFO:teuthology.orchestra.run.smithi102.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-25T07:02:52.307 INFO:teuthology.orchestra.run.smithi102.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-25T07:02:52.309 INFO:teuthology.orchestra.run.smithi102.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-25T07:02:52.346 INFO:teuthology.orchestra.run.smithi102.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-25T07:02:52.357 INFO:teuthology.orchestra.run.smithi102.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-25T07:02:52.357 INFO:teuthology.orchestra.run.smithi102.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-25T07:02:52.358 INFO:teuthology.orchestra.run.smithi102.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-25T07:02:52.389 INFO:teuthology.orchestra.run.smithi102.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-25T07:02:52.391 INFO:teuthology.orchestra.run.smithi102.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-25T07:02:52.395 INFO:teuthology.orchestra.run.smithi102.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-25T07:02:52.396 INFO:teuthology.orchestra.run.smithi102.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-25T07:02:52.399 INFO:teuthology.orchestra.run.smithi102.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-25T07:02:52.401 INFO:teuthology.orchestra.run.smithi102.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-25T07:02:52.432 INFO:teuthology.orchestra.run.smithi102.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-25T07:02:52.433 INFO:teuthology.orchestra.run.smithi102.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-25T07:02:52.434 INFO:teuthology.orchestra.run.smithi102.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-25T07:02:52.439 INFO:teuthology.orchestra.run.smithi102.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-25T07:02:52.534 INFO:teuthology.orchestra.run.smithi102.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3227-gc37f63b3-1jammy [112 kB] 2024-04-25T07:02:52.536 INFO:teuthology.orchestra.run.smithi102.stdout:Get:61 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3227-gc37f63b3-1jammy [466 kB] 2024-04-25T07:02:52.551 INFO:teuthology.orchestra.run.smithi102.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3227-gc37f63b3-1jammy [25.8 MB] 2024-04-25T07:02:52.592 INFO:teuthology.orchestra.run.smithi102.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-25T07:02:52.593 INFO:teuthology.orchestra.run.smithi102.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-25T07:02:52.594 INFO:teuthology.orchestra.run.smithi102.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-25T07:02:52.595 INFO:teuthology.orchestra.run.smithi102.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-25T07:02:52.596 INFO:teuthology.orchestra.run.smithi102.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-25T07:02:52.631 INFO:teuthology.orchestra.run.smithi102.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-25T07:02:52.635 INFO:teuthology.orchestra.run.smithi102.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-25T07:02:52.635 INFO:teuthology.orchestra.run.smithi102.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-25T07:02:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-25T07:02:52.668 INFO:teuthology.orchestra.run.smithi102.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-25T07:02:52.690 INFO:teuthology.orchestra.run.smithi102.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-25T07:02:52.691 INFO:teuthology.orchestra.run.smithi102.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-25T07:02:52.691 INFO:teuthology.orchestra.run.smithi102.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-25T07:02:52.692 INFO:teuthology.orchestra.run.smithi102.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-25T07:02:52.692 INFO:teuthology.orchestra.run.smithi102.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-25T07:02:52.693 INFO:teuthology.orchestra.run.smithi102.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-25T07:02:52.697 INFO:teuthology.orchestra.run.smithi102.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-25T07:02:52.727 INFO:teuthology.orchestra.run.smithi102.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-25T07:02:52.729 INFO:teuthology.orchestra.run.smithi102.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-25T07:02:52.729 INFO:teuthology.orchestra.run.smithi102.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-25T07:02:52.764 INFO:teuthology.orchestra.run.smithi102.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-25T07:02:52.769 INFO:teuthology.orchestra.run.smithi102.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-25T07:02:52.771 INFO:teuthology.orchestra.run.smithi102.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-25T07:02:52.772 INFO:teuthology.orchestra.run.smithi102.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-25T07:02:52.781 INFO:teuthology.orchestra.run.smithi102.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-25T07:02:52.782 INFO:teuthology.orchestra.run.smithi102.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-25T07:02:52.789 INFO:teuthology.orchestra.run.smithi102.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-25T07:02:52.809 INFO:teuthology.orchestra.run.smithi102.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-25T07:02:52.810 INFO:teuthology.orchestra.run.smithi102.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-25T07:02:52.810 INFO:teuthology.orchestra.run.smithi102.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-25T07:02:52.847 INFO:teuthology.orchestra.run.smithi102.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-25T07:02:52.849 INFO:teuthology.orchestra.run.smithi102.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-25T07:02:52.864 INFO:teuthology.orchestra.run.smithi102.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-25T07:02:52.865 INFO:teuthology.orchestra.run.smithi102.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-25T07:02:52.866 INFO:teuthology.orchestra.run.smithi102.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-25T07:02:52.871 INFO:teuthology.orchestra.run.smithi102.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-25T07:02:52.872 INFO:teuthology.orchestra.run.smithi102.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-25T07:02:53.549 INFO:teuthology.orchestra.run.smithi102.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3227-gc37f63b3-1jammy [5,092 kB] 2024-04-25T07:02:53.627 INFO:teuthology.orchestra.run.smithi102.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3227-gc37f63b3-1jammy [244 kB] 2024-04-25T07:02:53.630 INFO:teuthology.orchestra.run.smithi102.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3227-gc37f63b3-1jammy [124 kB] 2024-04-25T07:02:53.631 INFO:teuthology.orchestra.run.smithi102.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3227-gc37f63b3-1jammy [1,526 kB] 2024-04-25T07:02:53.661 INFO:teuthology.orchestra.run.smithi102.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3227-gc37f63b3-1jammy [6,257 kB] 2024-04-25T07:02:53.875 INFO:teuthology.orchestra.run.smithi102.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3227-gc37f63b3-1jammy [22.8 MB] 2024-04-25T07:02:54.723 INFO:teuthology.orchestra.run.smithi102.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3227-gc37f63b3-1jammy [14.1 kB] 2024-04-25T07:02:54.724 INFO:teuthology.orchestra.run.smithi102.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3227-gc37f63b3-1jammy [870 kB] 2024-04-25T07:02:54.741 INFO:teuthology.orchestra.run.smithi102.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3227-gc37f63b3-1jammy [2,437 kB] 2024-04-25T07:02:54.782 INFO:teuthology.orchestra.run.smithi102.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3227-gc37f63b3-1jammy [791 kB] 2024-04-25T07:02:54.791 INFO:teuthology.orchestra.run.smithi102.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3227-gc37f63b3-1jammy [148 kB] 2024-04-25T07:02:54.793 INFO:teuthology.orchestra.run.smithi102.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3227-gc37f63b3-1jammy [2,297 kB] 2024-04-25T07:02:54.838 INFO:teuthology.orchestra.run.smithi102.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3227-gc37f63b3-1jammy [8,625 kB] 2024-04-25T07:02:55.121 INFO:teuthology.orchestra.run.smithi102.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3227-gc37f63b3-1jammy [14.2 kB] 2024-04-25T07:02:55.122 INFO:teuthology.orchestra.run.smithi102.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3227-gc37f63b3-1jammy [51.6 MB] 2024-04-25T07:02:57.049 INFO:teuthology.orchestra.run.smithi102.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3227-gc37f63b3-1jammy [124 kB] 2024-04-25T07:02:57.052 INFO:teuthology.orchestra.run.smithi102.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3227-gc37f63b3-1jammy [39.0 kB] 2024-04-25T07:02:57.053 INFO:teuthology.orchestra.run.smithi102.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3227-gc37f63b3-1jammy [13.6 MB] 2024-04-25T07:02:57.388 INFO:teuthology.orchestra.run.smithi102.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3227-gc37f63b3-1jammy [91.4 kB] 2024-04-25T07:02:58.014 INFO:teuthology.orchestra.run.smithi102.stdout:Fetched 181 MB in 6s (30.3 MB/s) 2024-04-25T07:02:58.158 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-25T07:02:58.258 INFO:teuthology.orchestra.run.smithi102.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-25T07:02:58.263 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-25T07:02:58.331 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T07:02:58.651 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-25T07:02:58.678 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-25T07:02:58.718 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T07:02:59.062 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-25T07:02:59.089 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-25T07:02:59.129 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T07:02:59.508 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-25T07:02:59.534 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T07:02:59.583 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:02:59.994 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-25T07:03:00.021 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T07:03:00.061 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:03:00.414 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-25T07:03:00.441 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T07:03:00.481 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:03:00.792 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-25T07:03:00.819 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-25T07:03:00.850 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T07:03:01.153 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package librados2. 2024-04-25T07:03:01.180 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../007-librados2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:01.220 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking librados2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:01.698 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package librbd1. 2024-04-25T07:03:01.724 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../008-librbd1_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:01.764 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking librbd1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:02.192 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libcephfs2. 2024-04-25T07:03:02.219 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:02.259 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libcephfs2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:02.494 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-rados. 2024-04-25T07:03:02.510 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:02.544 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-rados (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:02.905 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-25T07:03:02.932 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:03:02.980 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-ceph-argparse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:03.233 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-cephfs. 2024-04-25T07:03:03.249 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:03.291 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-cephfs (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:03.577 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-25T07:03:03.604 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:03:03.644 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:03.964 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-25T07:03:03.990 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-25T07:03:04.030 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T07:03:04.359 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-prettytable. 2024-04-25T07:03:04.385 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-25T07:03:04.425 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-25T07:03:04.677 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-rbd. 2024-04-25T07:03:04.704 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:04.744 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-rbd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:05.297 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-25T07:03:05.324 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-25T07:03:05.363 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T07:03:05.633 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-25T07:03:05.659 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-25T07:03:05.699 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T07:03:06.027 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-25T07:03:06.054 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-25T07:03:06.094 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T07:03:06.397 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package lua5.1. 2024-04-25T07:03:06.424 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-25T07:03:06.455 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-25T07:03:06.767 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package lua-any. 2024-04-25T07:03:06.782 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-25T07:03:06.825 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-25T07:03:07.069 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package zip. 2024-04-25T07:03:07.085 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-25T07:03:07.127 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking zip (3.0-12build2) ... 2024-04-25T07:03:07.439 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package unzip. 2024-04-25T07:03:07.466 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-25T07:03:07.514 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-25T07:03:07.868 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package luarocks. 2024-04-25T07:03:07.895 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-25T07:03:07.934 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-25T07:03:08.309 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package librgw2. 2024-04-25T07:03:08.336 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../025-librgw2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:08.375 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking librgw2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:08.961 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-rgw. 2024-04-25T07:03:08.988 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:09.028 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-rgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:09.331 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-25T07:03:09.357 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-25T07:03:09.389 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T07:03:09.711 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libradosstriper1. 2024-04-25T07:03:09.737 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:09.777 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libradosstriper1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:10.206 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-common. 2024-04-25T07:03:10.233 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:10.272 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:11.627 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-base. 2024-04-25T07:03:11.654 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:11.703 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-base (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:12.298 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-25T07:03:12.325 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-25T07:03:12.365 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-25T07:03:12.676 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-cheroot. 2024-04-25T07:03:12.703 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-25T07:03:12.743 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T07:03:13.071 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-25T07:03:13.098 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-25T07:03:13.137 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-25T07:03:13.398 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-25T07:03:13.425 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-25T07:03:13.456 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-25T07:03:13.751 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-25T07:03:13.778 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-25T07:03:13.818 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-25T07:03:14.121 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-tempora. 2024-04-25T07:03:14.147 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-25T07:03:14.187 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-25T07:03:14.465 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-portend. 2024-04-25T07:03:14.492 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-25T07:03:14.531 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-25T07:03:14.801 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-25T07:03:14.828 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-25T07:03:14.867 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-25T07:03:15.204 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-25T07:03:15.231 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-25T07:03:15.395 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-25T07:03:15.749 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-natsort. 2024-04-25T07:03:15.776 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-25T07:03:15.807 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-25T07:03:16.119 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-logutils. 2024-04-25T07:03:16.146 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-25T07:03:16.185 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-25T07:03:16.488 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-25T07:03:16.515 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-25T07:03:16.554 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-25T07:03:16.858 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-mako. 2024-04-25T07:03:16.884 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-25T07:03:16.924 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T07:03:17.210 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-25T07:03:17.226 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-25T07:03:17.260 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-25T07:03:17.547 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-25T07:03:17.574 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-25T07:03:17.613 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-25T07:03:17.924 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-webob. 2024-04-25T07:03:17.951 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-25T07:03:17.991 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-25T07:03:18.386 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-waitress. 2024-04-25T07:03:18.412 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-25T07:03:18.455 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T07:03:18.730 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-tempita. 2024-04-25T07:03:18.756 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-25T07:03:18.796 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T07:03:19.041 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-paste. 2024-04-25T07:03:19.068 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-25T07:03:19.116 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T07:03:19.444 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-25T07:03:19.460 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-25T07:03:19.494 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T07:03:19.730 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-25T07:03:19.746 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-25T07:03:19.789 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-25T07:03:20.092 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-webtest. 2024-04-25T07:03:20.119 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-25T07:03:20.158 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-25T07:03:20.469 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pecan. 2024-04-25T07:03:20.496 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-25T07:03:20.536 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T07:03:20.908 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-certifi. 2024-04-25T07:03:20.934 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-25T07:03:20.974 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-25T07:03:21.302 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-idna. 2024-04-25T07:03:21.329 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-25T07:03:21.360 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-25T07:03:21.647 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-urllib3. 2024-04-25T07:03:21.673 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-25T07:03:21.713 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T07:03:21.991 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-requests. 2024-04-25T07:03:22.018 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-25T07:03:22.057 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T07:03:22.369 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-25T07:03:22.396 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-25T07:03:22.460 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T07:03:22.730 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-25T07:03:22.757 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:03:22.797 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:23.160 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-25T07:03:23.186 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:23.218 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:23.513 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-25T07:03:23.539 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-25T07:03:23.579 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-25T07:03:23.883 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-openssl. 2024-04-25T07:03:23.909 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-25T07:03:23.948 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-25T07:03:24.927 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr. 2024-04-25T07:03:24.955 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:25.169 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:25.522 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mon. 2024-04-25T07:03:25.549 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:25.588 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mon (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:26.409 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-osd. 2024-04-25T07:03:26.436 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:26.475 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-osd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:27.578 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph. 2024-04-25T07:03:27.594 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../066-ceph_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:27.628 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:27.890 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-fuse. 2024-04-25T07:03:27.917 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:27.956 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:28.317 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mds. 2024-04-25T07:03:28.344 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:28.384 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mds (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:28.779 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package cephadm. 2024-04-25T07:03:28.806 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../069-cephadm_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:28.845 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:29.123 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-25T07:03:29.150 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-25T07:03:29.181 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-25T07:03:29.525 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-25T07:03:29.541 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T07:03:29.584 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T07:03:30.622 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-babel. 2024-04-25T07:03:30.650 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T07:03:30.688 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T07:03:31.008 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-jinja2. 2024-04-25T07:03:31.035 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-25T07:03:31.074 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T07:03:31.445 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-25T07:03:31.472 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:03:31.511 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:31.864 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-25T07:03:31.892 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-25T07:03:31.931 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-25T07:03:32.234 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-routes. 2024-04-25T07:03:32.261 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-25T07:03:32.300 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T07:03:32.645 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-25T07:03:32.673 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:03:32.712 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:34.018 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-25T07:03:34.033 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-25T07:03:34.066 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T07:03:34.562 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-joblib. 2024-04-25T07:03:34.589 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-25T07:03:34.628 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T07:03:34.990 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-25T07:03:35.018 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-25T07:03:35.056 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-25T07:03:35.343 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-sklearn. 2024-04-25T07:03:35.370 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-25T07:03:35.401 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T07:03:36.118 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-25T07:03:36.146 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:03:36.286 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:37.239 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-cachetools. 2024-04-25T07:03:37.256 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-25T07:03:37.289 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-25T07:03:37.551 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-25T07:03:37.591 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-25T07:03:37.617 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-25T07:03:37.921 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-25T07:03:37.948 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-25T07:03:37.987 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T07:03:38.298 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-rsa. 2024-04-25T07:03:38.326 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-25T07:03:38.365 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-25T07:03:38.685 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-google-auth. 2024-04-25T07:03:38.713 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-25T07:03:38.752 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-25T07:03:39.080 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-25T07:03:39.108 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-25T07:03:39.146 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T07:03:39.383 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-websocket. 2024-04-25T07:03:39.399 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-25T07:03:39.432 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-25T07:03:39.736 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-25T07:03:39.764 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-25T07:03:39.802 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T07:03:40.467 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-25T07:03:40.494 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:03:40.533 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:40.785 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-25T07:03:40.814 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-25T07:03:40.860 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T07:03:41.205 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-25T07:03:41.232 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T07:03:41.272 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T07:03:41.758 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package jq. 2024-04-25T07:03:41.773 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T07:03:41.816 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-25T07:03:42.052 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package socat. 2024-04-25T07:03:42.068 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-25T07:03:42.110 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-25T07:03:42.422 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package xmlstarlet. 2024-04-25T07:03:42.437 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-25T07:03:42.480 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-25T07:03:42.834 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-test. 2024-04-25T07:03:42.861 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:42.900 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-test (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:45.197 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package ceph-volume. 2024-04-25T07:03:45.224 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:03:45.263 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking ceph-volume (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:45.592 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-25T07:03:45.619 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:45.658 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking libcephfs-dev (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:45.937 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-25T07:03:45.964 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-25T07:03:46.130 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T07:03:46.507 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-25T07:03:46.535 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-25T07:03:46.574 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-25T07:03:46.945 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package nvme-cli. 2024-04-25T07:03:46.972 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-25T07:03:47.011 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T07:03:47.433 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package pkg-config. 2024-04-25T07:03:47.460 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-25T07:03:47.499 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T07:03:47.727 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-25T07:03:47.743 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-25T07:03:47.814 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-25T07:03:48.206 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-25T07:03:48.233 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-25T07:03:48.263 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-25T07:03:48.533 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pastescript. 2024-04-25T07:03:48.560 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-25T07:03:48.599 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-25T07:03:48.928 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pluggy. 2024-04-25T07:03:48.955 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-25T07:03:48.996 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-25T07:03:49.333 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-psutil. 2024-04-25T07:03:49.360 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-25T07:03:49.399 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-25T07:03:49.760 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-py. 2024-04-25T07:03:49.788 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-25T07:03:49.835 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-25T07:03:50.139 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pygments. 2024-04-25T07:03:50.166 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-25T07:03:50.205 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T07:03:50.651 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-25T07:03:50.678 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-25T07:03:50.709 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-25T07:03:51.012 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-toml. 2024-04-25T07:03:51.039 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-25T07:03:51.078 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-25T07:03:51.390 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-pytest. 2024-04-25T07:03:51.417 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-25T07:03:51.456 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T07:03:51.793 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package python3-simplejson. 2024-04-25T07:03:51.820 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-25T07:03:51.859 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-25T07:03:52.148 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-25T07:03:52.164 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-25T07:03:52.211 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-25T07:03:52.916 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package radosgw. 2024-04-25T07:03:52.943 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../116-radosgw_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:52.982 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking radosgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:53.810 INFO:teuthology.orchestra.run.smithi102.stdout:Selecting previously unselected package rbd-fuse. 2024-04-25T07:03:53.838 INFO:teuthology.orchestra.run.smithi102.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:53.877 INFO:teuthology.orchestra.run.smithi102.stdout:Unpacking rbd-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:54.181 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-25T07:03:54.415 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T07:03:54.516 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T07:03:54.755 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-25T07:03:55.073 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-25T07:03:55.487 INFO:teuthology.orchestra.run.smithi102.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-25T07:03:55.560 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:56.085 INFO:teuthology.orchestra.run.smithi102.stdout:Adding system user cephadm....done 2024-04-25T07:03:56.250 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T07:03:56.736 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-25T07:03:56.990 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-25T07:03:57.430 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-25T07:03:57.689 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-25T07:03:57.948 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T07:03:58.058 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-25T07:03:58.360 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T07:03:58.720 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-25T07:03:58.963 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-25T07:03:59.231 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-25T07:03:59.474 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-25T07:03:59.717 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-25T07:03:59.985 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-25T07:04:00.253 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-ceph-argparse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:00.512 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T07:04:00.613 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T07:04:00.731 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T07:04:00.857 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-25T07:04:01.116 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-25T07:04:01.530 INFO:teuthology.orchestra.run.smithi102.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-25T07:04:01.588 INFO:teuthology.orchestra.run.smithi102.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-25T07:04:01.653 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T07:04:01.762 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-25T07:04:02.113 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T07:04:02.214 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-25T07:04:02.474 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T07:04:02.733 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-25T07:04:03.043 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up zip (3.0-12build2) ... 2024-04-25T07:04:03.169 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T07:04:03.854 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T07:04:04.097 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T07:04:04.198 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-25T07:04:04.299 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T07:04:04.592 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-25T07:04:04.977 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-25T07:04:05.228 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T07:04:05.588 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T07:04:05.873 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T07:04:06.216 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-25T07:04:06.467 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-25T07:04:06.593 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:06.894 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-idna (3.3-1) ... 2024-04-25T07:04:07.203 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T07:04:08.401 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T07:04:08.661 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:04:08.795 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-25T07:04:09.047 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T07:04:09.173 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-25T07:04:09.299 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T07:04:09.609 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-25T07:04:09.868 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-25T07:04:10.119 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:04:10.229 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-25T07:04:10.513 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-25T07:04:10.781 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-25T07:04:11.057 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-25T07:04:11.317 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-25T07:04:11.568 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-25T07:04:11.827 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T07:04:11.937 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-25T07:04:12.204 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T07:04:12.314 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T07:04:12.607 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-25T07:04:12.908 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-25T07:04:13.167 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T07:04:13.293 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-25T07:04:13.419 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T07:04:13.500 INFO:teuthology.orchestra.run.smithi102.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-25T07:04:13.737 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T07:04:13.855 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T07:04:14.249 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-25T07:04:14.508 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-25T07:04:14.634 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-25T07:04:14.885 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:04:15.011 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T07:04:15.320 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T07:04:15.613 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T07:04:15.898 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-25T07:04:16.024 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-25T07:04:16.300 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-25T07:04:16.584 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T07:04:16.937 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-25T07:04:17.306 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-25T07:04:17.594 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T07:04:17.927 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T07:04:18.044 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up librados2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:18.154 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:18.263 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-25T07:04:18.608 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libcephfs2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:18.734 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libradosstriper1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:18.860 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-25T07:04:19.162 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up librbd1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:19.271 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr-modules-core (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:19.381 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:19.595 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T07:04:19.596 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T07:04:20.065 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up libcephfs-dev (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:20.191 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-rados (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:20.317 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up librgw2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:20.460 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T07:04:21.664 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-rbd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:21.790 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up rbd-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:21.916 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-rgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:22.042 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up python3-cephfs (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:22.168 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:22.660 INFO:teuthology.orchestra.run.smithi102.stdout:Adding group ceph....done 2024-04-25T07:04:23.062 INFO:teuthology.orchestra.run.smithi102.stdout:Adding system user ceph....done 2024-04-25T07:04:23.222 INFO:teuthology.orchestra.run.smithi102.stdout:Setting system user ceph properties....done 2024-04-25T07:04:23.237 INFO:teuthology.orchestra.run.smithi102.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-25T07:04:23.374 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-25T07:04:23.666 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-25T07:04:24.199 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-test (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:24.384 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up radosgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:24.951 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T07:04:24.951 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T07:04:25.457 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-base (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:25.779 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-25T07:04:26.251 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mds (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:26.476 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T07:04:26.476 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T07:04:26.961 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:27.182 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T07:04:27.182 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T07:04:27.662 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-osd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:27.929 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T07:04:27.929 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T07:04:28.433 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:28.559 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:28.710 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mon (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:28.950 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T07:04:28.950 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T07:04:29.403 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr-cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:29.538 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:29.664 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-mgr-dashboard (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:29.815 INFO:teuthology.orchestra.run.smithi102.stdout:Setting up ceph-volume (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:30.185 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T07:04:30.311 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T07:04:30.455 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T07:04:31.247 INFO:teuthology.orchestra.run.smithi102.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T07:04:31.885 INFO:teuthology.orchestra.run.smithi102.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T07:04:31.890 DEBUG:teuthology.parallel:result is None 2024-04-25T07:04:31.890 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T07:04:32.071 DEBUG:teuthology.orchestra.run.smithi102:> dpkg-query -W -f '${Version}' ceph 2024-04-25T07:04:32.086 INFO:teuthology.orchestra.run.smithi102.stdout:19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:04:32.086 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:04:32.087 INFO:teuthology.task.install:The correct ceph version 19.0.0-3227-gc37f63b3-1jammy is installed. 2024-04-25T07:04:32.088 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T07:04:32.089 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T07:04:32.089 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T07:04:32.143 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T07:04:32.143 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T07:04:32.143 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T07:04:32.198 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T07:04:32.254 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T07:04:32.255 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T07:04:32.255 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T07:04:32.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T07:04:32.365 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T07:04:32.366 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T07:04:32.366 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T07:04:32.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T07:04:32.475 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T07:04:32.489 INFO:tasks.workunit:Pulling workunits from ref a08304379f921e5e229e92c75ede8800059313ff 2024-04-25T07:04:32.489 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T07:04:32.489 INFO:tasks.workunit:timeout=3h 2024-04-25T07:04:32.489 INFO:tasks.workunit:cleanup=True 2024-04-25T07:04:32.489 DEBUG:teuthology.orchestra.run.smithi102:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T07:04:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T07:04:32.524 INFO:teuthology.orchestra.run.smithi102.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-25T07:04:32.524 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T07:04:32.571 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-25T07:04:32.571 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-25T07:04:32.619 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/NitzanMordhai/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a08304379f921e5e229e92c75ede8800059313ff 2024-04-25T07:04:32.668 INFO:tasks.workunit.client.0.smithi102.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T07:05:11.048 INFO:tasks.workunit.client.0.smithi102.stderr:Note: switching to 'a08304379f921e5e229e92c75ede8800059313ff'. 2024-04-25T07:05:11.048 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T07:05:11.048 INFO:tasks.workunit.client.0.smithi102.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T07:05:11.048 INFO:tasks.workunit.client.0.smithi102.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr: git switch -c 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr:Or undo this operation with: 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr: git switch - 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T07:05:11.049 INFO:tasks.workunit.client.0.smithi102.stderr:HEAD is now at a08304379f9 Merge pull request #56420 from rzarzynski/wip-tools-rados-get-with-offset 2024-04-25T07:05:11.051 DEBUG:teuthology.orchestra.run.smithi102:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T07:05:11.099 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-04-25T07:05:11.099 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T07:05:11.146 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-04-25T07:05:11.146 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-04-25T07:05:11.146 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-04-25T07:05:11.209 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:05:11.239 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/bad-inc-map 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:39527 2024-04-25T07:05:11.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:39527 2024-04-25T07:05:11.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-04-25T07:05:11.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-04-25T07:05:11.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=1c626bfc-685a-4a90-a0dc-8d9a466b2fe2 --auth-supported=none ' 2024-04-25T07:05:11.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:39527 ' 2024-04-25T07:05:11.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-04-25T07:05:11.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-04-25T07:05:11.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:05:11.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-04-25T07:05:11.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-04-25T07:05:11.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-04-25T07:05:11.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2024-04-25T07:05:11.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2024-04-25T07:05:11.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T07:05:11.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:05:11.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T07:05:11.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:05:11.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:05:11.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:05:11.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:05:11.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:05:11.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:05:11.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:05:11.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:05:11.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:05:11.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:05:11.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:05:11.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:05:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:05:11.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:05:11.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:11.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:05:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:05:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T07:05:11.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:05:11.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:11.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20295 2024-04-25T07:05:11.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20295 2024-04-25T07:05:11.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:05:11.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:05:11.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2024-04-25T07:05:11.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:05:11.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:11.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20295 2024-04-25T07:05:11.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20295 2024-04-25T07:05:11.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:05:11.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:05:11.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:05:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:05:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-04-25T07:05:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-04-25T07:05:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-04-25T07:05:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-04-25T07:05:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2024-04-25T07:05:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:05:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:05:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:05:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2024-04-25T07:05:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-04-25T07:05:11.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:05:11.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:11.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:11.901 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:11.901 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:11.901 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20295 2024-04-25T07:05:11.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20295/$cluster-$name.asok' 2024-04-25T07:05:11.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.20295/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:05:12.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:05:12.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:05:12.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:05:12.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:05:12.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:05:12.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:05:12.137 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:05:12.137 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:05:12.137 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:05:12.138 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:05:12.138 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:12.138 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20295 2024-04-25T07:05:12.138 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20295/ceph-mon.a.asok 2024-04-25T07:05:12.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:05:12.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20295/ceph-mon.a.asok config get fsid 2024-04-25T07:05:12.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:05:12.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:05:12.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:05:12.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:05:12.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:05:12.264 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:05:12.264 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:05:12.264 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:05:12.265 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:05:12.265 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:12.266 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20295 2024-04-25T07:05:12.266 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20295/ceph-mon.a.asok 2024-04-25T07:05:12.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:05:12.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20295/ceph-mon.a.asok config get mon_host 2024-04-25T07:05:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-04-25T07:05:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2024-04-25T07:05:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:05:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:05:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:05:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2024-04-25T07:05:12.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:05:13.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:05:13.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:13.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:13.255 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:13.255 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:13.255 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20295 2024-04-25T07:05:13.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20295/$cluster-$name.asok' 2024-04-25T07:05:13.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:05:13.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.20295/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:05:13.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-04-25T07:05:13.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T07:05:13.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:05:13.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:05:13.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1c626bfc-685a-4a90-a0dc-8d9a466b2fe2 --auth-supported=none --mon-host=127.0.0.1:39527 ' 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:13.413 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20295 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20295/$cluster-$name.asok' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20295/$cluster-$name.asok' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:05:13.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:05:13.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:05:13.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2024-04-25T07:05:13.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:05:13.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e6fbb20b-d954-44ce-93c0-d491a21ee140 2024-04-25T07:05:13.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e6fbb20b-d954-44ce-93c0-d491a21ee140' 2024-04-25T07:05:13.416 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 e6fbb20b-d954-44ce-93c0-d491a21ee140 2024-04-25T07:05:13.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:05:13.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpACpmn+PPGRAAzoCiZchMXsqMnTjdDYYdKQ== 2024-04-25T07:05:13.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpACpmn+PPGRAAzoCiZchMXsqMnTjdDYYdKQ=="}' 2024-04-25T07:05:13.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e6fbb20b-d954-44ce-93c0-d491a21ee140 -i td/bad-inc-map/0/new.json 2024-04-25T07:05:13.713 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:05:13.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2024-04-25T07:05:13.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1c626bfc-685a-4a90-a0dc-8d9a466b2fe2 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20295/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpACpmn+PPGRAAzoCiZchMXsqMnTjdDYYdKQ== --osd-uuid e6fbb20b-d954-44ce-93c0-d491a21ee140 2024-04-25T07:05:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:13.825+0000 7fb7bf9897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:13.829 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:13.825+0000 7fb7bf9897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:13.830 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:13.829+0000 7fb7bf9897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:13.831 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:13.829+0000 7fb7bf9897c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T07:05:13.831 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:13.829+0000 7fb7bf9897c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T07:05:13.831 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:13.829+0000 7fb7bf9897c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T07:05:13.931 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:13.929+0000 7fb7bf9897c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-04-25T07:05:19.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-04-25T07:05:19.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:05:19.974 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:05:19.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:05:19.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:05:20.214 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:05:20.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:05:20.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:05:20.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1c626bfc-685a-4a90-a0dc-8d9a466b2fe2 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20295/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:05:20.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:05:20.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:05:20.245 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:20.241+0000 7f2f6b9447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:20.248 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:20.245+0000 7f2f6b9447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:20.249 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:20.249+0000 7f2f6b9447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:20.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:05:20.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:05:20.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:05:20.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:05:20.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:05:20.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:20.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:05:20.436 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:05:20.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:20.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:20.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:20.796 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:20.793+0000 7f2f6b9447c0 -1 Falling back to public interface 2024-04-25T07:05:21.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:21.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:21.652 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:05:21.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:05:21.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:21.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:21.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:22.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:05:22.943 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:05:22.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:22.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:23.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:23.523 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:23.521+0000 7f2f6b9447c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:05:24.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:24.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:24.364 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:05:24.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:05:24.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:24.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:24.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:25.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:25.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:25.904 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:05:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:05:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:25.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:26.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:27.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:27.422 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:05:27.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:27.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:05:27.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:27.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:27.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:28.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:28.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:28.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:05:28.827 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:05:28.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:28.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:29.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:30.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:30.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:30.365 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:05:30.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:05:30.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:30.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:30.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:31.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:31.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:31.819 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:05:31.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:05:31.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:31.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:32.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:33.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:33.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:33.395 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:05:33.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:05:33.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:33.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:33.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:34.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:34.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:34.803 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:05:34.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:05:34.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:34.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:35.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:36.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:36.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:36.379 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:05:36.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:05:36.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:36.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:36.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:37.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:37.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:37.901 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:05:37.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:05:37.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:37.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:38.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:39.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:39.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:39.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:05:39.523 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:05:39.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:39.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:39.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:40.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:40.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:40.972 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:05:40.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:05:40.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:40.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:41.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:42.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:42.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:42.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:05:42.472 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:05:42.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:42.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:43.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:44.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:44.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:44.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:05:44.264 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:05:44.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:44.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:46.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:47.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:47.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:47.026 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:05:47.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:05:47.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:47.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:48.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:49.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:49.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:49.407 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:05:49.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:05:49.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:49.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:05:50.056 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3479098747,v1:127.0.0.1:6801/3479098747] [v2:127.0.0.1:6802/3479098747,v1:127.0.0.1:6803/3479098747] exists,up e6fbb20b-d954-44ce-93c0-d491a21ee140 2024-04-25T07:05:50.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1c626bfc-685a-4a90-a0dc-8d9a466b2fe2 --auth-supported=none --mon-host=127.0.0.1:39527 ' 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:05:50.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20295 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20295/$cluster-$name.asok' 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20295/$cluster-$name.asok' 2024-04-25T07:05:50.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:05:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:05:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:05:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T07:05:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T07:05:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:05:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:05:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:05:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:05:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:05:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:05:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2024-04-25T07:05:50.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:05:50.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=29dd275c-6256-4ecd-bb1c-62ae6236be22 2024-04-25T07:05:50.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 29dd275c-6256-4ecd-bb1c-62ae6236be22' 2024-04-25T07:05:50.063 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 29dd275c-6256-4ecd-bb1c-62ae6236be22 2024-04-25T07:05:50.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:05:50.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDOACpmUtqxBBAA/auCGxWTvPYFA191WKhxQA== 2024-04-25T07:05:50.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDOACpmUtqxBBAA/auCGxWTvPYFA191WKhxQA=="}' 2024-04-25T07:05:50.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 29dd275c-6256-4ecd-bb1c-62ae6236be22 -i td/bad-inc-map/1/new.json 2024-04-25T07:05:50.602 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:05:50.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2024-04-25T07:05:50.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1c626bfc-685a-4a90-a0dc-8d9a466b2fe2 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20295/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDOACpmUtqxBBAA/auCGxWTvPYFA191WKhxQA== --osd-uuid 29dd275c-6256-4ecd-bb1c-62ae6236be22 2024-04-25T07:05:50.651 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:50.649+0000 7fc8384a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:50.708 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:50.709+0000 7fc8384a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:50.710 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:50.709+0000 7fc8384a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:50.710 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:50.709+0000 7fc8384a57c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T07:05:50.710 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:50.709+0000 7fc8384a57c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T07:05:50.710 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:50.709+0000 7fc8384a57c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T07:05:50.827 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:50.829+0000 7fc8384a57c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-04-25T07:05:57.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-04-25T07:05:57.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:05:57.355 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:05:57.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:05:57.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:05:57.940 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:05:57.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:05:57.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:05:57.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:05:57.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1c626bfc-685a-4a90-a0dc-8d9a466b2fe2 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20295/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:05:57.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:05:57.969 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:57.969+0000 7f05414cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:57.991 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:57.989+0000 7f05414cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:57.992 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:57.989+0000 7f05414cd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:58.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:05:58.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:05:58.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:05:58.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:05:58.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:05:58.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:58.408 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:05:58.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:05:58.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:58.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:05:58.567 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:05:58.565+0000 7f05414cd7c0 -1 Falling back to public interface 2024-04-25T07:05:58.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:05:59.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:05:59.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:05:59.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:05:59.854 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:05:59.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:05:59.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:00.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:01.199 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:01.197+0000 7f05414cd7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:06:01.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:01.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:01.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:06:01.255 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:06:01.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:01.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:02.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:03.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:03.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:03.055 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:06:03.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:06:03.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:03.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:03.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:04.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:04.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:04.508 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:06:04.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:06:04.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:04.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:05.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:06.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:06.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:06.007 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:06:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:06:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:06.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:07.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:07.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:07.530 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:06:07.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:06:07.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:07.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:08.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:09.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:09.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:09.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:06:09.167 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:06:09.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:09.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:09.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:10.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:10.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:10.591 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:06:10.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:06:10.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:10.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:11.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:12.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:12.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:12.182 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:06:12.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:06:12.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:12.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:12.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:13.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:13.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:13.583 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:06:13.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:06:13.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:13.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:14.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:15.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:15.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:15.153 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:06:15.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:06:15.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:15.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:15.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:16.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:16.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:16.709 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:06:16.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:06:16.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:16.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:17.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:18.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:18.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:18.583 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:06:18.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:06:18.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:18.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:19.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:20.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:20.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:20.780 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:06:20.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:06:20.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:20.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:21.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:22.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:22.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:22.524 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:06:22.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:06:22.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:22.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:23.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:24.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:24.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:24.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:06:24.381 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:06:24.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:24.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:25.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:26.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:26.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:26.424 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:06:26.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:06:26.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:26.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:26.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:27.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:27.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:27.829 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:06:27.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:06:27.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:27.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:28.277 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3145725946,v1:127.0.0.1:6811/3145725946] [v2:127.0.0.1:6812/3145725946,v1:127.0.0.1:6813/3145725946] exists,up 29dd275c-6256-4ecd-bb1c-62ae6236be22 2024-04-25T07:06:28.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:06:28.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:06:28.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:06:28.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-04-25T07:06:28.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T07:06:28.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:06:28.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:06:28.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:06:28.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2024-04-25T07:06:28.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1c626bfc-685a-4a90-a0dc-8d9a466b2fe2 --auth-supported=none --mon-host=127.0.0.1:39527 ' 2024-04-25T07:06:28.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:06:28.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20295 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20295/$cluster-$name.asok' 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20295/$cluster-$name.asok' 2024-04-25T07:06:28.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:06:28.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:06:28.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:06:28.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T07:06:28.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T07:06:28.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:06:28.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:06:28.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:06:28.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:06:28.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:06:28.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:06:28.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2024-04-25T07:06:28.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:06:28.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6fe91172-6ad8-4bff-b95c-082ceb3bd5bb 2024-04-25T07:06:28.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6fe91172-6ad8-4bff-b95c-082ceb3bd5bb' 2024-04-25T07:06:28.284 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 6fe91172-6ad8-4bff-b95c-082ceb3bd5bb 2024-04-25T07:06:28.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:06:28.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0ACpmEyoaEhAAgK5BmHoh2YVccFve3cfm7w== 2024-04-25T07:06:28.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0ACpmEyoaEhAAgK5BmHoh2YVccFve3cfm7w=="}' 2024-04-25T07:06:28.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6fe91172-6ad8-4bff-b95c-082ceb3bd5bb -i td/bad-inc-map/2/new.json 2024-04-25T07:06:28.772 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:06:28.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2024-04-25T07:06:28.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1c626bfc-685a-4a90-a0dc-8d9a466b2fe2 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20295/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0ACpmEyoaEhAAgK5BmHoh2YVccFve3cfm7w== --osd-uuid 6fe91172-6ad8-4bff-b95c-082ceb3bd5bb 2024-04-25T07:06:28.820 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:28.819+0000 7f9bc68c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:28.822 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:28.823+0000 7f9bc68c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:28.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:28.823+0000 7f9bc68c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:28.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:28.823+0000 7f9bc68c97c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T07:06:28.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:28.823+0000 7f9bc68c97c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T07:06:28.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:28.823+0000 7f9bc68c97c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T07:06:28.909 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:28.907+0000 7f9bc68c97c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-04-25T07:06:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-04-25T07:06:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:06:35.466 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:06:35.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:06:35.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:06:36.044 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:06:36.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:06:36.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1c626bfc-685a-4a90-a0dc-8d9a466b2fe2 --auth-supported=none --mon-host=127.0.0.1:39527 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20295/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:06:36.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:06:36.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:06:36.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:06:36.073 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:36.071+0000 7f8ba19557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:36.099 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:36.099+0000 7f8ba19557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:36.101 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:36.099+0000 7f8ba19557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:36.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:06:36.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:06:36.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:06:36.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:06:36.531 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:06:36.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:06:36.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:36.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:06:36.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:36.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:36.667 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:36.667+0000 7f8ba19557c0 -1 Falling back to public interface 2024-04-25T07:06:36.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:37.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:37.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:37.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:06:37.947 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:06:37.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:37.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:39.298 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:06:39.295+0000 7f8ba19557c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:06:39.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:39.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:39.347 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:06:39.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:06:39.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:39.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:39.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:40.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:40.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:40.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:06:40.869 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:06:40.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:40.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:41.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:42.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:42.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:42.364 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:06:42.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:06:42.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:42.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:42.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:43.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:43.896 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:06:43.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:43.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:06:43.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:43.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:44.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:45.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:45.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:45.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:06:45.440 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:06:45.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:45.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:45.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:46.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:46.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:46.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:06:46.876 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:06:46.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:46.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:47.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:48.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:48.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:48.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:06:48.349 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:06:48.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:48.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:48.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:49.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:49.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:49.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:06:49.951 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:06:49.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:49.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:50.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:51.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:51.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:51.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:06:51.490 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:06:51.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:51.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:51.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:52.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:52.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:52.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:06:52.916 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:06:52.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:52.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:54.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:55.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:55.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:55.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:06:55.013 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:06:55.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:55.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:55.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:56.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:56.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:56.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:06:56.709 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:06:56.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:56.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:57.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:58.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:58.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:58.826 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:06:58.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:06:58.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:58.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:06:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:00.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:00.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:00.806 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:07:00.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:07:00.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:00.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:01.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:02.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:02.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:02.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:07:02.607 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:07:02.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:02.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:03.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:04.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:04.471 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:07:04.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:04.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:07:04.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:04.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:04.917 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2486922395,v1:127.0.0.1:6819/2486922395] [v2:127.0.0.1:6820/2486922395,v1:127.0.0.1:6821/2486922395] exists,up 6fe91172-6ad8-4bff-b95c-082ceb3bd5bb 2024-04-25T07:07:04.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:07:04.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:07:04.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:07:04.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-04-25T07:07:05.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-04-25T07:07:05.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2024-04-25T07:07:06.133 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T07:07:06.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:07:07.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-04-25T07:07:08.180 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 min_size to 1 2024-04-25T07:07:08.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-04-25T07:07:09.800 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 min_size to 2 2024-04-25T07:07:09.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-04-25T07:07:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T07:07:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-04-25T07:07:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:07:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:07:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:07:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:07:14.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T07:07:14.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:07:14.823 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:07:14.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:14.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:15.239 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3479098747,v1:127.0.0.1:6801/3479098747] [v2:127.0.0.1:6802/3479098747,v1:127.0.0.1:6803/3479098747] exists,up e6fbb20b-d954-44ce-93c0-d491a21ee140 2024-04-25T07:07:15.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:07:15.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:07:15.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:07:15.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T07:07:15.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-04-25T07:07:15.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:07:15.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:07:15.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:07:15.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:07:15.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T07:07:15.240 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:07:15.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:07:15.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:07:15.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:15.648 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3145725946,v1:127.0.0.1:6811/3145725946] [v2:127.0.0.1:6812/3145725946,v1:127.0.0.1:6813/3145725946] exists,up 29dd275c-6256-4ecd-bb1c-62ae6236be22 2024-04-25T07:07:15.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:07:15.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:07:15.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:07:15.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T07:07:15.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-04-25T07:07:15.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:07:15.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:07:15.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:07:15.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:07:15.649 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:07:15.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T07:07:15.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:07:15.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:15.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:16.051 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2486922395,v1:127.0.0.1:6819/2486922395] [v2:127.0.0.1:6820/2486922395,v1:127.0.0.1:6821/2486922395] exists,up 6fe91172-6ad8-4bff-b95c-082ceb3bd5bb 2024-04-25T07:07:16.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:07:16.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:07:16.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:07:16.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:05.958+0000 7f8b928f9640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:06.606+0000 7f8b928f9640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:07.006+0000 7f8b928f9640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:08.006+0000 7f8b908f5640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:08.406+0000 7f8b908f5640 -1 osd.2 20 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:08.914+0000 7f8b928f9640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:09.614+0000 7f8b908f5640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.034+0000 7f8b908f5640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.210+0000 7f8b908f5640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.406+0000 7f8b928f9640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.458+0000 7f8b928f9640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.846+0000 7f8b928f9640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.054 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.850+0000 7f8b928f9640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-04-25T07:07:16.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-04-25T07:07:16.056 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:05.958+0000 7f8b928f9640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-04-25T07:07:16.056 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:06.606+0000 7f8b928f9640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-04-25T07:07:16.056 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:07.006+0000 7f8b928f9640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-04-25T07:07:16.056 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:08.006+0000 7f8b908f5640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-04-25T07:07:16.056 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:08.406+0000 7f8b908f5640 10 osd.2 20 handle_osd_map bailing because last < start (20<21) 2024-04-25T07:07:16.056 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:08.914+0000 7f8b928f9640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-04-25T07:07:16.057 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:09.618+0000 7f8b908f5640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-04-25T07:07:16.057 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.034+0000 7f8b908f5640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-04-25T07:07:16.057 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.210+0000 7f8b908f5640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-04-25T07:07:16.057 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.406+0000 7f8b928f9640 10 osd.2 23 handle_osd_map bailing because last < start (24<25) 2024-04-25T07:07:16.057 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.458+0000 7f8b928f9640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-04-25T07:07:16.057 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.846+0000 7f8b928f9640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-04-25T07:07:16.057 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:07:10.850+0000 7f8b928f9640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-04-25T07:07:16.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-04-25T07:07:16.057 INFO:tasks.workunit.client.0.smithi102.stdout:success 2024-04-25T07:07:16.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-04-25T07:07:16.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2024-04-25T07:07:16.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-04-25T07:07:16.565 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' does not exist 2024-04-25T07:07:16.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-04-25T07:07:16.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:07:16.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:07:16.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:07:16.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:07:16.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:07:21.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:07:21.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-04-25T07:07:21.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T07:07:21.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:07:21.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T07:07:21.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:07:21.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:07:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:07:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:07:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:07:21.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:07:21.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:07:21.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:07:21.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:07:21.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:07:21.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:07:21.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:07:21.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:07:21.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:07:21.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:07:21.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:07:21.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:07:21.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:07:21.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T07:07:21.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:07:21.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:21.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20295 2024-04-25T07:07:21.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20295 2024-04-25T07:07:21.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:07:21.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:07:21.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:07:21.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/bad-inc-map 0 2024-04-25T07:07:21.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T07:07:21.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:07:21.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T07:07:21.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:07:21.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:07:21.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:07:21.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:07:21.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:07:21.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:07:21.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:07:21.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:07:21.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:07:21.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:07:21.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:07:21.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:07:21.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:07:21.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:07:21.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:07:21.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:07:21.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:07:21.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:07:21.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T07:07:21.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:07:21.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:21.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20295 2024-04-25T07:07:21.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20295 2024-04-25T07:07:21.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:07:21.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:07:21.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:07:21.943 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:07:21.943 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:07:21.997 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-04-25T07:07:21.997 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-04-25T07:07:22.051 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:07:22.058 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:07:22.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:07:22.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:07:22.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:07:22.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:07:22.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:07:22.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:07:22.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:07:22.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:07:22.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/divergent-priors 2024-04-25T07:07:22.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-04-25T07:07:22.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-04-25T07:07:22.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-04-25T07:07:22.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-04-25T07:07:22.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-04-25T07:07:22.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-04-25T07:07:22.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-04-25T07:07:22.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-04-25T07:07:22.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-04-25T07:07:22.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-04-25T07:07:22.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-04-25T07:07:22.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-04-25T07:07:22.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-04-25T07:07:22.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-04-25T07:07:22.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-04-25T07:07:22.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-04-25T07:07:22.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-04-25T07:07:22.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-04-25T07:07:22.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-04-25T07:07:22.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-04-25T07:07:22.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-04-25T07:07:22.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-04-25T07:07:22.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T07:07:22.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-04-25T07:07:22.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-04-25T07:07:22.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-04-25T07:07:22.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none ' 2024-04-25T07:07:22.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T07:07:22.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T07:07:22.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-04-25T07:07:22.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:07:22.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-04-25T07:07:22.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:07:22.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-04-25T07:07:22.072 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_divergent_2 2024-04-25T07:07:22.072 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_divergent_3 2024-04-25T07:07:22.072 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_divergent_ec' 2024-04-25T07:07:22.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:07:22.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:07:22.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:07:22.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:07:22.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:07:22.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:07:22.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:07:22.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:07:22.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:07:22.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:07:22.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:07:22.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:07:22.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:07:22.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:07:22.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:07:22.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:07:22.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:07:22.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:07:22.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:07:22.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:07:22.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:07:22.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:07:22.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:07:22.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:07:22.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:07:22.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:07:22.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:07:22.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:22.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:07:22.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26156 2024-04-25T07:07:22.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:07:22.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:07:22.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:07:22.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:07:22.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:22.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:07:22.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26156 2024-04-25T07:07:22.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:07:22.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:07:22.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:07:22.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:07:22.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:07:22.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-04-25T07:07:22.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-04-25T07:07:22.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-04-25T07:07:22.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-04-25T07:07:22.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-04-25T07:07:22.086 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-04-25T07:07:22.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-04-25T07:07:22.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-04-25T07:07:22.096 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:07:22.096 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:07:22.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-04-25T07:07:22.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:07:22.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:07:22.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:07:22.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:07:22.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:07:22.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:07:22.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:07:22.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:07:22.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:07:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:07:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:07:22.616 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:07:22.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:07:22.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:07:22.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:07:22.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:07:22.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:07:22.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:07:22.836 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:07:22.836 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:07:22.836 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:07:22.837 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:07:22.837 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:07:22.837 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:22.837 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:07:22.837 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-mon.a.asok 2024-04-25T07:07:22.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:07:22.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26156/ceph-mon.a.asok config get fsid 2024-04-25T07:07:22.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:07:22.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:07:22.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:07:22.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:07:22.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:07:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:07:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:07:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:07:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:07:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:07:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-mon.a.asok 2024-04-25T07:07:22.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:07:22.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26156/ceph-mon.a.asok config get mon_host 2024-04-25T07:07:23.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-04-25T07:07:23.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:07:23.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:07:23.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:07:23.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:07:23.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:07:23.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:07:23.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:07:23.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:07:23.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:07:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:07:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:23.963 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:07:23.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:07:23.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:07:23.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:07:24.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T07:07:24.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-04-25T07:07:24.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:07:24.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:07:24.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:07:24.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:07:24.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:07:24.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:07:24.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:07:24.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0936f745-b82d-4ca4-9a42-d371e0fa2286 2024-04-25T07:07:24.009 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 0936f745-b82d-4ca4-9a42-d371e0fa2286 2024-04-25T07:07:24.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0936f745-b82d-4ca4-9a42-d371e0fa2286' 2024-04-25T07:07:24.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:07:24.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsASpmXw2pARAAhQNMad1o12zcOO3PslNKcg== 2024-04-25T07:07:24.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsASpmXw2pARAAhQNMad1o12zcOO3PslNKcg=="}' 2024-04-25T07:07:24.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0936f745-b82d-4ca4-9a42-d371e0fa2286 -i td/divergent-priors/0/new.json 2024-04-25T07:07:24.330 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:07:24.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:07:24.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsASpmXw2pARAAhQNMad1o12zcOO3PslNKcg== --osd-uuid 0936f745-b82d-4ca4-9a42-d371e0fa2286 2024-04-25T07:07:24.382 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:24.378+0000 7fd61b4a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:24.398 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:24.394+0000 7fd61b4a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:24.399 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:24.398+0000 7fd61b4a97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:24.399 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:24.398+0000 7fd61b4a97c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:07:24.399 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:24.398+0000 7fd61b4a97c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:07:24.400 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:24.398+0000 7fd61b4a97c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:07:24.429 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:24.426+0000 7fd61b4a97c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:07:30.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:07:30.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:07:30.594 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:07:30.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:07:30.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:07:31.158 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:07:31.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:07:31.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:07:31.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:07:31.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:07:31.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:07:31.197 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:31.194+0000 7fe9438fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:31.206 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:31.202+0000 7fe9438fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:31.207 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:31.206+0000 7fe9438fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:31.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:07:31.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:07:31.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:07:31.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:07:31.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:07:31.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:31.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:07:31.571 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:07:31.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:31.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:31.784 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:31.782+0000 7fe9438fc7c0 -1 Falling back to public interface 2024-04-25T07:07:31.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:32.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:32.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:32.974 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:07:32.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:07:32.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:32.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:33.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:34.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:07:34.354+0000 7fe9438fc7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:07:34.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:34.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:34.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:07:34.427 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:07:34.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:34.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:35.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:36.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:36.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:36.005 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:07:36.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:07:36.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:36.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:36.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:37.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:37.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:37.489 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:07:37.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:07:37.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:37.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:37.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:38.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:38.942 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:07:38.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:38.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:07:38.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:38.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:39.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:40.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:40.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:40.520 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:07:40.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:07:40.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:40.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:40.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:41.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:41.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:41.940 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:07:41.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:07:41.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:41.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:42.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:43.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:43.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:43.454 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:07:43.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:07:43.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:43.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:43.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:44.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:44.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:44.903 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:07:44.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:07:44.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:44.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:45.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:46.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:46.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:46.471 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:07:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:07:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:46.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:47.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:47.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:47.919 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:07:47.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:07:47.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:47.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:48.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:49.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:49.772 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:07:49.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:49.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:07:49.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:49.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:50.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:51.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:51.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:51.564 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:07:51.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:07:51.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:51.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:52.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:53.436 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:07:53.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:07:53.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:53.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:54.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:55.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:55.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:55.578 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:07:55.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:07:55.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:55.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:56.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:57.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:57.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:57.429 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:07:57.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:07:57.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:57.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:58.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:59.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:59.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:59.278 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:07:59.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:07:59.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:59.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:59.798 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/634908359,v1:127.0.0.1:6803/634908359] [v2:127.0.0.1:6804/634908359,v1:127.0.0.1:6805/634908359] exists,up 0936f745-b82d-4ca4-9a42-d371e0fa2286 2024-04-25T07:07:59.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:07:59.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:07:59.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:07:59.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T07:07:59.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-04-25T07:07:59.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:07:59.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:07:59.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:07:59.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:07:59.801 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:07:59.802 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:59.802 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:07:59.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:07:59.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:07:59.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=056e4b63-b68a-492b-83e9-1ebaac50e9c4 2024-04-25T07:07:59.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 056e4b63-b68a-492b-83e9-1ebaac50e9c4' 2024-04-25T07:07:59.805 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 056e4b63-b68a-492b-83e9-1ebaac50e9c4 2024-04-25T07:07:59.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:07:59.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPASpmWBaZMRAAJGNCDjYh4kq3f9TIvFwtxg== 2024-04-25T07:07:59.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPASpmWBaZMRAAJGNCDjYh4kq3f9TIvFwtxg=="}' 2024-04-25T07:07:59.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 056e4b63-b68a-492b-83e9-1ebaac50e9c4 -i td/divergent-priors/1/new.json 2024-04-25T07:08:00.321 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:08:00.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:08:00.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPASpmWBaZMRAAJGNCDjYh4kq3f9TIvFwtxg== --osd-uuid 056e4b63-b68a-492b-83e9-1ebaac50e9c4 2024-04-25T07:08:00.378 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:00.373+0000 7fed5143a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:00.406 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:00.405+0000 7fed5143a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:00.408 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:00.405+0000 7fed5143a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:00.409 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:00.405+0000 7fed5143a7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:08:00.409 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:00.405+0000 7fed5143a7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:08:00.409 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:00.405+0000 7fed5143a7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:08:00.446 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:00.445+0000 7fed5143a7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:08:06.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:08:06.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:08:06.849 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:08:06.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:08:06.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:08:07.411 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:08:07.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:08:07.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:08:07.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:08:07.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:08:07.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:08:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:07.449+0000 7f67989587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:07.470 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:07.469+0000 7f67989587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:07.471 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:07.469+0000 7f67989587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:07.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:08:07.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:08:07.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:08:07.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:08:07.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:08:07.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:07.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:08:07.852 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:08:07.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:07.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:08.047 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:08.045+0000 7f67989587c0 -1 Falling back to public interface 2024-04-25T07:08:08.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:09.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:09.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:09.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:08:09.296 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:08:09.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:09.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:09.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:10.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:10.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:10.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:08:10.736 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:08:10.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:10.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:10.766 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:10.761+0000 7f67989587c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:08:11.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:12.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:12.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:12.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:08:12.326 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:08:12.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:12.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:12.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:13.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:13.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:13.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:08:13.856 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:08:13.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:13.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:14.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:15.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:15.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:15.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:08:15.278 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:08:15.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:15.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:15.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:16.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:16.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:16.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:08:16.915 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:08:16.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:16.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:17.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:18.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:18.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:18.419 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:08:18.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:08:18.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:18.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:19.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:20.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:20.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:20.035 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:08:20.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:08:20.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:20.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:20.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:21.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:21.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:21.462 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:08:21.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:08:21.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:21.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:22.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:23.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:23.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:23.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:08:23.050 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:08:23.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:23.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:24.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:24.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:24.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:08:24.551 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:08:24.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:24.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:25.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:26.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:26.395 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:08:26.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:08:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:27.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:28.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:28.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:28.210 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:08:28.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:08:28.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:28.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:28.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:29.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:29.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:29.983 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:08:29.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:08:29.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:29.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:30.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:31.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:31.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:31.772 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:08:31.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:08:31.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:31.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:32.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:33.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:33.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:33.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:08:33.911 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:08:33.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:33.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:34.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:35.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:35.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:35.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:08:35.857 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:08:35.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:35.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:36.322 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3876753015,v1:127.0.0.1:6811/3876753015] [v2:127.0.0.1:6812/3876753015,v1:127.0.0.1:6813/3876753015] exists,up 056e4b63-b68a-492b-83e9-1ebaac50e9c4 2024-04-25T07:08:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:08:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:08:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:08:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:08:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:08:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:08:36.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:08:36.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:08:36.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:08:36.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:08:36.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:08:36.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:08:36.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:08:36.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:08:36.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:08:36.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a15e78f2-04f0-4690-aca5-f26e6ea79b64 2024-04-25T07:08:36.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a15e78f2-04f0-4690-aca5-f26e6ea79b64' 2024-04-25T07:08:36.330 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 a15e78f2-04f0-4690-aca5-f26e6ea79b64 2024-04-25T07:08:36.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:08:36.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0ASpmUA5cFRAA0GqfPhFkY7ceRn3es8qxYA== 2024-04-25T07:08:36.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0ASpmUA5cFRAA0GqfPhFkY7ceRn3es8qxYA=="}' 2024-04-25T07:08:36.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a15e78f2-04f0-4690-aca5-f26e6ea79b64 -i td/divergent-priors/2/new.json 2024-04-25T07:08:36.874 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:08:36.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:08:36.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0ASpmUA5cFRAA0GqfPhFkY7ceRn3es8qxYA== --osd-uuid a15e78f2-04f0-4690-aca5-f26e6ea79b64 2024-04-25T07:08:36.934 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:36.933+0000 7fc07031c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:36.937 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:36.933+0000 7fc07031c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:36.938 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:36.937+0000 7fc07031c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:36.938 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:36.937+0000 7fc07031c7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:08:36.939 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:36.937+0000 7fc07031c7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:08:36.939 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:36.937+0000 7fc07031c7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:08:36.982 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:36.981+0000 7fc07031c7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:08:43.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:08:43.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:08:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:08:43.637 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:08:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:08:44.205 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:08:44.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:08:44.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:08:44.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:08:44.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:08:44.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:08:44.239 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:44.236+0000 7ff6489157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:44.265 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:44.260+0000 7ff6489157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:44.267 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:44.264+0000 7ff6489157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:44.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:08:44.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:08:44.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:08:44.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:08:44.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:08:44.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:44.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:08:44.650 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:08:44.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:44.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:44.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:44.820+0000 7ff6489157c0 -1 Falling back to public interface 2024-04-25T07:08:45.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:46.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:46.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:46.069 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:08:46.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:08:46.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:46.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:46.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:47.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:47.491 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:08:47.488+0000 7ff6489157c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:08:47.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:47.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:08:47.491 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:08:47.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:47.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:48.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:49.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:49.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:49.006 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:08:49.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:08:49.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:49.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:49.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:50.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:50.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:50.448 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:08:50.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:08:50.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:50.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:50.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:51.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:51.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:51.934 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:08:51.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:08:51.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:51.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:52.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:53.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:53.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:53.509 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:08:53.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:08:53.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:53.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:54.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:55.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:55.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:55.090 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:08:55.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:08:55.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:55.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:55.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:56.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:56.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:56.760 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:08:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:08:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:57.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:58.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:58.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:58.341 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:08:58.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:08:58.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:58.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:08:58.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:59.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:59.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:59.787 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:08:59.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:08:59.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:59.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:00.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:01.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:01.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:01.246 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:09:01.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:09:01.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:01.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:02.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:03.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:03.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:03.082 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:09:03.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:09:03.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:03.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:04.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:05.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:05.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:05.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:09:05.155 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:09:05.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:05.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:05.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:06.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:06.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:06.977 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:09:06.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:09:06.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:06.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:08.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:09.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:09.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:09.152 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:09:09.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:09:09.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:09.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:09.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:10.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:10.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:10.943 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:09:10.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:09:10.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:10.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:11.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:12.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:12.855 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:09:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:09:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:12.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:13.284 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2226734851,v1:127.0.0.1:6819/2226734851] [v2:127.0.0.1:6820/2226734851,v1:127.0.0.1:6821/2226734851] exists,up a15e78f2-04f0-4690-aca5-f26e6ea79b64 2024-04-25T07:09:13.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:09:13.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:09:13.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:09:13.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-04-25T07:09:14.082 INFO:tasks.workunit.client.0.smithi102.stderr:noout is set 2024-04-25T07:09:14.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-04-25T07:09:15.061 INFO:tasks.workunit.client.0.smithi102.stderr:noin is set 2024-04-25T07:09:15.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-04-25T07:09:15.932 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T07:09:15.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-04-25T07:09:15.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T07:09:16.681 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T07:09:16.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:09:17.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-04-25T07:09:18.377 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 3 2024-04-25T07:09:18.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-04-25T07:09:19.197 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 min_size to 2 2024-04-25T07:09:19.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-04-25T07:09:19.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:09:19.218 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:09:19.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:09:19.698 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:09:19.698 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:09:19.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:09:19.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:19.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:09:19.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836506 2024-04-25T07:09:19.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836506 2024-04-25T07:09:19.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506' 2024-04-25T07:09:19.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:19.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:09:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672979 2024-04-25T07:09:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672979 2024-04-25T07:09:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672979' 2024-04-25T07:09:20.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:20.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:09:20.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509449 2024-04-25T07:09:20.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509449 2024-04-25T07:09:20.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672979 2-64424509449' 2024-04-25T07:09:20.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:09:20.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836506 2024-04-25T07:09:20.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:09:20.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:09:20.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-04-25T07:09:20.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:09:20.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836506 2024-04-25T07:09:20.176 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836506 2024-04-25T07:09:20.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-04-25T07:09:20.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:09:20.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836506 2024-04-25T07:09:20.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:09:21.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:09:21.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:09:22.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836506 2024-04-25T07:09:22.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:09:22.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672979 2024-04-25T07:09:22.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:09:22.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:09:22.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:09:22.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672979 2024-04-25T07:09:22.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672979 2024-04-25T07:09:22.081 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672979 2024-04-25T07:09:22.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-04-25T07:09:22.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:09:22.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672979 2024-04-25T07:09:22.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:09:22.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509449 2024-04-25T07:09:22.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:09:22.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:09:22.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509449 2024-04-25T07:09:22.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:09:22.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509449 2024-04-25T07:09:22.509 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509449 2024-04-25T07:09:22.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-04-25T07:09:22.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:09:22.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509449 2024-04-25T07:09:22.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-04-25T07:09:22.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:09:22.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:09:22.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:09:22.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:09:22.968 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:09:22.968 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:09:22.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:09:22.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:09:22.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:09:23.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:09:23.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:09:23.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:09:23.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:09:23.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:09:23.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:09:23.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:09:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:09:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:09:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:09:23.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:23.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:09:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836508 2024-04-25T07:09:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836508 2024-04-25T07:09:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508' 2024-04-25T07:09:23.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:23.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:09:23.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672982 2024-04-25T07:09:23.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672982 2024-04-25T07:09:23.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672982' 2024-04-25T07:09:23.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:23.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:09:23.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509452 2024-04-25T07:09:23.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509452 2024-04-25T07:09:23.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672982 2-64424509452' 2024-04-25T07:09:23.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:09:23.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836508 2024-04-25T07:09:23.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:09:23.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:09:23.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-04-25T07:09:23.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:09:23.987 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836508 2024-04-25T07:09:23.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836508 2024-04-25T07:09:23.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-04-25T07:09:23.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:09:24.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836508 2024-04-25T07:09:24.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:09:25.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:09:25.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:09:25.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836508 2024-04-25T07:09:25.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:09:25.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672982 2024-04-25T07:09:25.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:09:25.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:09:25.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-04-25T07:09:25.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:09:25.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672982 2024-04-25T07:09:25.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-04-25T07:09:25.881 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672982 2024-04-25T07:09:25.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:09:26.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672982 2024-04-25T07:09:26.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:09:26.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509452 2024-04-25T07:09:26.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:09:26.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:09:26.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509452 2024-04-25T07:09:26.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:09:26.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509452 2024-04-25T07:09:26.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-04-25T07:09:26.303 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509452 2024-04-25T07:09:26.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:09:26.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509452 -lt 64424509452 2024-04-25T07:09:26.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:09:26.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:26.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:27.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:09:27.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:09:27.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:09:27.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:09:27.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:09:27.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:09:27.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:09:27.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:09:27.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:09:27.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:09:27.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:27.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:28.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:09:28.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:09:28.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:09:28.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-04-25T07:09:28.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-04-25T07:09:28.695 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:09:28.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-04-25T07:09:28.711 INFO:tasks.workunit.client.0.smithi102.stdout:primary and soon to be divergent is 1 2024-04-25T07:09:28.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-04-25T07:09:28.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-04-25T07:09:29.097 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:29.097 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:09:17.510190+0000 0'0 29:19 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:42:42.877753+0000 0 0 2024-04-25T07:09:29.097 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:09:29.098 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:29.098 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:09:29.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-04-25T07:09:29.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T07:09:29.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-04-25T07:09:29.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-04-25T07:09:29.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T07:09:29.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-04-25T07:09:29.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-04-25T07:09:29.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T07:09:29.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-04-25T07:09:29.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-04-25T07:09:29.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-04-25T07:09:29.114 INFO:tasks.workunit.client.0.smithi102.stdout:writing initial objects 2024-04-25T07:09:29.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-04-25T07:09:29.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:29.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-04-25T07:09:29.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:29.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-04-25T07:09:29.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:29.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-04-25T07:09:29.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:29.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-04-25T07:09:29.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:29.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-04-25T07:09:30.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:30.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-04-25T07:09:30.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:30.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-04-25T07:09:30.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:30.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-04-25T07:09:30.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:30.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-04-25T07:09:30.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:30.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-04-25T07:09:30.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:30.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-04-25T07:09:31.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:31.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-04-25T07:09:31.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:31.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-04-25T07:09:31.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:31.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-04-25T07:09:31.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:31.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-04-25T07:09:31.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:31.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-04-25T07:09:32.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:32.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-04-25T07:09:32.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:32.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-04-25T07:09:32.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:32.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-04-25T07:09:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-04-25T07:09:32.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:32.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-04-25T07:09:32.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:32.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-04-25T07:09:33.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:33.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-04-25T07:09:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:33.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-04-25T07:09:33.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:33.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-04-25T07:09:33.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:33.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-04-25T07:09:33.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:33.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-04-25T07:09:33.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:33.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-04-25T07:09:34.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:34.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-04-25T07:09:34.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:34.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-04-25T07:09:34.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:34.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-04-25T07:09:34.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:34.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-04-25T07:09:34.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:34.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-04-25T07:09:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-04-25T07:09:35.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:35.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-04-25T07:09:35.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:35.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-04-25T07:09:35.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:35.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-04-25T07:09:35.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:35.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-04-25T07:09:35.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:35.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-04-25T07:09:36.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:36.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-04-25T07:09:36.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:36.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-04-25T07:09:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:36.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-04-25T07:09:36.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:36.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-04-25T07:09:36.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:36.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-04-25T07:09:36.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:36.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-04-25T07:09:37.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:37.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-04-25T07:09:37.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:37.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-04-25T07:09:37.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:37.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-04-25T07:09:37.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:37.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-04-25T07:09:37.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:37.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-04-25T07:09:38.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:38.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-04-25T07:09:38.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:38.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-04-25T07:09:38.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:38.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-04-25T07:09:38.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:38.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-04-25T07:09:38.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:38.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-04-25T07:09:38.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:38.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-04-25T07:09:39.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:39.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-04-25T07:09:39.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:39.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-04-25T07:09:39.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:39.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-04-25T07:09:39.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:39.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-04-25T07:09:39.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:39.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-04-25T07:09:40.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:40.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-04-25T07:09:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-04-25T07:09:40.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:40.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-04-25T07:09:40.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:40.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-04-25T07:09:40.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:40.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-04-25T07:09:40.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:40.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-04-25T07:09:41.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:41.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-04-25T07:09:41.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:41.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-04-25T07:09:41.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:41.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-04-25T07:09:41.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:41.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-04-25T07:09:41.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:41.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-04-25T07:09:42.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:42.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-04-25T07:09:42.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:42.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-04-25T07:09:42.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:42.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-04-25T07:09:42.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:42.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-04-25T07:09:42.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:42.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-04-25T07:09:42.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:42.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-04-25T07:09:43.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:43.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-04-25T07:09:43.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:43.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-04-25T07:09:43.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:43.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-04-25T07:09:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:43.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-04-25T07:09:43.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:43.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-04-25T07:09:44.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:44.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-04-25T07:09:44.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:44.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-04-25T07:09:44.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:44.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-04-25T07:09:44.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:44.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-04-25T07:09:44.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:44.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-04-25T07:09:44.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:44.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-04-25T07:09:45.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:45.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-04-25T07:09:45.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:45.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-04-25T07:09:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-04-25T07:09:45.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:45.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-04-25T07:09:45.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:45.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-04-25T07:09:46.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:46.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-04-25T07:09:46.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:46.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-04-25T07:09:46.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:46.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-04-25T07:09:46.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:46.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-04-25T07:09:46.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:46.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-04-25T07:09:46.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:09:46.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-04-25T07:09:47.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:09:47.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-04-25T07:09:47.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:09:47.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:09:47.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:09:47.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:09:47.149 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:09:47.149 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:09:47.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:09:47.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:09:47.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:09:47.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:09:47.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:09:47.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:09:47.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:09:47.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:09:47.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:09:47.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:09:47.661 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:09:47.661 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:09:47.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:09:47.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:47.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:09:47.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836515 2024-04-25T07:09:47.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836515 2024-04-25T07:09:47.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515' 2024-04-25T07:09:47.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:47.838 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:09:47.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672989 2024-04-25T07:09:47.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672989 2024-04-25T07:09:47.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515 1-42949672989' 2024-04-25T07:09:47.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:09:47.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:09:48.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509458 2024-04-25T07:09:48.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509458 2024-04-25T07:09:48.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836515 1-42949672989 2-64424509458' 2024-04-25T07:09:48.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:09:48.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836515 2024-04-25T07:09:48.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:09:48.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:09:48.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836515 2024-04-25T07:09:48.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:09:48.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836515 2024-04-25T07:09:48.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-04-25T07:09:48.161 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836515 2024-04-25T07:09:48.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:09:48.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836513 -lt 21474836515 2024-04-25T07:09:48.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:09:49.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:09:49.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:09:49.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836515 -lt 21474836515 2024-04-25T07:09:49.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:09:49.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672989 2024-04-25T07:09:49.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:09:50.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:09:50.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672989 2024-04-25T07:09:50.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:09:50.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672989 2024-04-25T07:09:50.003 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672989 2024-04-25T07:09:50.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-04-25T07:09:50.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:09:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672989 -lt 42949672989 2024-04-25T07:09:50.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:09:50.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509458 2024-04-25T07:09:50.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:09:50.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:09:50.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-04-25T07:09:50.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:09:50.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509458 2024-04-25T07:09:50.439 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509458 2024-04-25T07:09:50.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-04-25T07:09:50.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:09:50.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509459 -lt 64424509458 2024-04-25T07:09:50.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:09:50.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:50.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:51.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:09:51.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:09:51.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:09:51.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:09:51.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:09:51.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:09:51.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:09:51.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:09:51.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:09:51.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:09:51.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:09:51.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:09:52.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:09:52.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:09:52.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:09:52.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-04-25T07:09:52.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:09:52.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:09:52.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:09:52.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:09:52.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-04-25T07:09:52.824 INFO:tasks.workunit.client.0.smithi102.stdout:blackholing osds 0 2 2024-04-25T07:09:52.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-04-25T07:09:52.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-04-25T07:09:53.208 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:09:53.208 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T07:09:17.510190+0000 29'100 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:42:42.877753+0000 0 0 2024-04-25T07:09:53.208 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:09:53.209 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:09:53.209 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:09:53.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-04-25T07:09:53.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-04-25T07:09:53.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:09:53.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:09:53.226 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:09:53.226 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:53.226 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:09:53.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.0.asok 2024-04-25T07:09:53.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-04-25T07:09:53.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:09:53.327 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:09:53.327 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:09:53.327 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:09:53.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-04-25T07:09:53.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-04-25T07:09:53.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:09:53.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:09:53.339 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:09:53.339 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:53.339 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:09:53.340 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.2.asok 2024-04-25T07:09:53.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-04-25T07:09:53.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:09:53.434 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:09:53.434 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:09:53.434 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:09:53.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-04-25T07:09:53.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-04-25T07:09:53.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-04-25T07:09:53.447 INFO:tasks.workunit.client.0.smithi102.stdout:writing divergent object 2024-04-25T07:09:53.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-04-25T07:09:53.448 INFO:tasks.workunit.client.0.smithi102.stdout:create missing divergent object 2024-04-25T07:09:53.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-04-25T07:09:53.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-04-25T07:09:53.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-04-25T07:09:53.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T07:09:53.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T07:09:53.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T07:09:53.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T07:09:53.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test 2024-04-25T07:09:53.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T07:09:53.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=existing_99 2024-04-25T07:09:53.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T07:09:53.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/divergent-priors 2024-04-25T07:09:53.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T07:09:53.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T07:09:53.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T07:09:53.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test existing_99 2024-04-25T07:09:53.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-04-25T07:09:53.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2024-04-25T07:09:53.451 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T07:09:53.451 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2024-04-25T07:09:53.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-25T07:09:53.929 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-04-25T07:09:53.929 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:09:53.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2024-04-25T07:09:53.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0' '2') 2024-04-25T07:09:53.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T07:09:53.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T07:09:53.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T07:09:53.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T07:09:53.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/divergent-priors/1/type 2024-04-25T07:09:53.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T07:09:53.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T07:09:53.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T07:09:53.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T07:09:53.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T07:09:53.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T07:09:53.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T07:09:53.932 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T07:09:53.932 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:09:53.932 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:09:53.933 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:09:53.933 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:53.933 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:09:53.933 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.1.asok 2024-04-25T07:09:53.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.26156/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T07:09:54.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T07:09:54.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T07:09:54.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T07:09:54.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T07:09:54.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:09:54.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:09:54.056 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:09:54.056 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:54.056 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:09:54.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.1.asok 2024-04-25T07:09:54.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T07:09:54.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.1.asok injectdataerr test existing_99 2024-04-25T07:09:54.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-04-25T07:09:54.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-04-25T07:10:04.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-04-25T07:10:04.195 INFO:tasks.workunit.client.0.smithi102.stderr:rados: no process found 2024-04-25T07:10:04.195 INFO:tasks.workunit.client.0.smithi102.stdout:killing all the osds 2024-04-25T07:10:04.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-04-25T07:10:04.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-04-25T07:10:04.597 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:10:04.597 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 active+clean 2024-04-25T07:09:54.360830+0000 29'101 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:42:42.877753+0000 0 0 2024-04-25T07:10:04.597 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:10:04.597 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:10:04.598 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:10:04.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:10:04.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:10:04.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:10:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:10:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:10:04.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:10:04.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:10:04.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T07:10:04.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-04-25T07:10:05.234 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T07:10:05.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T07:10:05.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-04-25T07:10:05.802 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already down. 2024-04-25T07:10:05.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T07:10:05.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-04-25T07:10:06.378 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already down. 2024-04-25T07:10:06.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:06.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-04-25T07:10:07.085 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. 2024-04-25T07:10:07.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:07.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-04-25T07:10:07.715 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already out. 2024-04-25T07:10:07.733 INFO:tasks.workunit.client.0.smithi102.stdout:bringing up non_divergent 0 2 2024-04-25T07:10:07.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-04-25T07:10:07.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-04-25T07:10:08.163 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:10:08.164 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 stale+active+clean 2024-04-25T07:09:54.360830+0000 29'101 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:42:42.877753+0000 0 0 2024-04-25T07:10:08.164 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:10:08.164 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:10:08.164 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:10:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-04-25T07:10:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:10:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:10:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:10:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:10:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:10:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:10:08.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:10:08.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:10:08.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:10:08.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:10:08.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:10:08.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:10:08.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:10:08.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:10:08.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:10:08.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:10:08.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:10:08.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:10:08.186 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:10:08.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:10:08.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:10:08.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:10:08.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:10:08.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:10:08.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:10:08.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:10:08.230 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:08.231+0000 7f293d69a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:08.233 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:08.231+0000 7f293d69a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:08.234 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:08.235+0000 7f293d69a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:08.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:10:08.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:10:08.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:10:08.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:10:08.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:10:08.589 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:10:08.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:08.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:10:08.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:08.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:08.806 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:08.803+0000 7f293d69a7c0 -1 Falling back to public interface 2024-04-25T07:10:09.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:10.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:10.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:10.089 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:10:10.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:10:10.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:10.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:10.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:11.572 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:10:11.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:10:11.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:11.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:11.847 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:11.847+0000 7f293d69a7c0 -1 osd.0 29 log_to_monitors true 2024-04-25T07:10:11.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:12.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:12.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:12.996 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:10:12.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:10:12.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:12.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:13.419 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1254124613,v1:127.0.0.1:6803/1254124613] [v2:127.0.0.1:6804/1254124613,v1:127.0.0.1:6805/1254124613] exists,up 0936f745-b82d-4ca4-9a42-d371e0fa2286 2024-04-25T07:10:13.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:10:13.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:10:13.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:10:13.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:13.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-04-25T07:10:13.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:10:13.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:10:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:10:13.426 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:10:13.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:10:13.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:10:13.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:10:13.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:10:13.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:10:13.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:10:13.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:10:13.463 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:13.463+0000 7f282c8c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:13.465 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:13.463+0000 7f282c8c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:13.466 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:13.467+0000 7f282c8c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:13.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:10:13.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:10:13.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:10:13.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:10:13.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:10:13.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:13.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:10:13.836 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:10:13.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:13.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:14.039 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:14.039+0000 7f282c8c87c0 -1 Falling back to public interface 2024-04-25T07:10:14.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:15.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:15.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:15.269 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:10:15.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:10:15.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:15.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:15.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:16.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:16.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:16.684 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:10:16.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:10:16.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:16.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:17.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:17.151 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:17.151+0000 7f282c8c87c0 -1 osd.2 29 log_to_monitors true 2024-04-25T07:10:18.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:18.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:18.126 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:10:18.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:10:18.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:18.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:18.555 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/919078068,v1:127.0.0.1:6811/919078068] [v2:127.0.0.1:6812/919078068,v1:127.0.0.1:6813/919078068] exists,up a15e78f2-04f0-4690-aca5-f26e6ea79b64 2024-04-25T07:10:18.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:10:18.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:10:18.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:10:18.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:18.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-04-25T07:10:19.092 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already in. 2024-04-25T07:10:19.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:19.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-04-25T07:10:19.649 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already in. 2024-04-25T07:10:19.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:10:19.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-04-25T07:10:19.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:10:19.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:10:19.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:10:19.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:10:19.665 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:10:19.665 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:10:19.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:10:19.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:10:19.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:10:19.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:10:19.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:10:19.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:10:19.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:10:19.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:10:19.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:10:20.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:10:20.215 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:10:20.215 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:10:20.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:10:20.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:20.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:10:20.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822660 2024-04-25T07:10:20.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822660 2024-04-25T07:10:20.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660' 2024-04-25T07:10:20.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:20.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:10:20.532 INFO:tasks.workunit.client.0.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:10:20.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:10:20.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:10:20.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:10:20.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:20.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:10:20.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757251 2024-04-25T07:10:20.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757251 2024-04-25T07:10:20.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-04-25T07:10:20.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:10:20.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-154618822660 2024-04-25T07:10:20.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:10:20.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:10:20.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:10:20.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822660 2024-04-25T07:10:20.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822660 2024-04-25T07:10:20.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-04-25T07:10:20.706 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 154618822660 2024-04-25T07:10:20.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:10:21.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822660 -lt 154618822660 2024-04-25T07:10:21.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:10:21.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-163208757251 2024-04-25T07:10:21.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:10:21.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:10:21.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757251 2024-04-25T07:10:21.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:10:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757251 2024-04-25T07:10:21.153 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 163208757251 2024-04-25T07:10:21.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-04-25T07:10:21.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:10:21.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757249 -lt 163208757251 2024-04-25T07:10:21.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:10:22.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:10:22.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:10:23.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757251 -lt 163208757251 2024-04-25T07:10:23.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:10:23.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:23.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:23.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:10:23.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:10:23.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:10:23.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:10:23.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:10:23.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:10:23.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:10:23.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:10:24.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:10:24.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:10:24.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:24.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:24.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:10:24.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:10:24.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:10:24.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:10:24.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:10:24.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:10:24.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:10:24.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:10:24.738 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:10:24.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:10:24.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:10:24.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:10:24.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:10:24.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:10:25.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:10:25.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:10:25.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:25.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:25.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:10:25.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:10:25.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:10:25.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:10:25.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:10:25.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:10:25.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:10:25.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:10:25.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:10:26.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:10:26.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:10:26.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:10:26.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:10:26.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:10:26.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:10:26.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:10:26.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:10:26.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:10:26.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:10:26.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:10:26.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:10:26.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:10:26.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:10:26.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:10:26.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:26.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:27.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:10:27.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:10:27.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:10:27.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:10:27.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:10:27.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:10:27.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:10:27.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:10:27.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:10:27.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:10:27.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:10:27.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:10:27.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:10:27.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:10:28.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:10:28.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:10:28.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:10:28.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:10:28.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:10:28.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:10:28.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:10:28.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:10:28.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:10:28.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:10:28.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:28.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:10:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:10:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:10:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:10:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:10:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:10:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:10:29.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:10:29.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:10:29.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:10:29.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:10:29.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:10:29.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:10:29.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:10:30.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:10:30.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:10:30.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:10:30.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:10:30.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:10:30.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:10:30.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:10:30.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:10:31.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:10:31.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:10:31.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:31.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:31.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:10:31.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:10:31.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:10:31.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:10:31.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:10:31.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:10:31.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:10:31.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:10:31.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:10:32.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:10:32.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:10:32.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:10:32.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:10:32.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:10:33.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:10:33.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:10:33.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:10:33.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:10:33.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:10:33.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:10:33.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:10:33.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:10:34.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:10:34.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:10:34.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:34.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:34.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:10:34.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:10:34.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:10:34.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:10:34.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:10:34.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:10:34.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:10:34.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:10:34.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:10:35.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:10:35.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:10:35.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:10:35.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:10:35.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:10:38.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:10:38.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:10:38.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:10:38.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:10:38.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:10:38.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:10:38.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:10:38.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:10:38.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:10:38.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:10:38.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:38.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:39.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:10:39.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:10:39.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:10:39.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:10:39.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:10:39.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:10:39.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:10:39.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:10:39.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:10:40.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:10:40.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:10:40.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:10:40.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:10:40.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:10:46.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:10:46.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:10:46.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:10:46.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:10:46.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:10:46.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:10:46.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:10:46.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:10:46.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:10:46.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:10:46.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:46.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:47.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:10:47.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:10:47.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:10:47.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:10:47.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:10:47.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:10:47.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:10:47.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:10:47.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:10:47.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:10:47.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:10:47.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:10:47.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:10:47.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:10:55.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:10:55.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:10:55.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:10:55.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:10:55.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:10:55.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:10:55.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:10:55.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:10:55.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:10:55.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:10:55.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:55.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:56.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:10:56.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:10:56.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:10:56.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:10:56.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:10:56.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:10:56.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:10:56.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:10:56.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:10:56.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:10:56.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:10:56.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:10:56.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:10:57.473 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:10:57.474 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_fingerprint": "10695b14-5cab-4f4c-ad8c-c7713b914f5c", 2024-04-25T07:10:57.474 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:10:57.474 INFO:tasks.workunit.client.0.smithi102.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:10:57.474 INFO:tasks.workunit.client.0.smithi102.stdout: "timestamp": "2024-04-25T07:10:57.472855+0000", 2024-04-25T07:10:57.474 INFO:tasks.workunit.client.0.smithi102.stdout: "tag": "", 2024-04-25T07:10:57.474 INFO:tasks.workunit.client.0.smithi102.stdout: "health": { 2024-04-25T07:10:57.474 INFO:tasks.workunit.client.0.smithi102.stdout: "status": "HEALTH_WARN", 2024-04-25T07:10:57.474 INFO:tasks.workunit.client.0.smithi102.stdout: "checks": { 2024-04-25T07:10:57.474 INFO:tasks.workunit.client.0.smithi102.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 17 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [], 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: "OSD_DOWN": { 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 osds down", 2024-04-25T07:10:57.475 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "osd.1 (root=default,host=smithi102) is down" 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: "PG_DEGRADED": { 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:10:57.476 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:10:57.477 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.478 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.479 INFO:tasks.workunit.client.0.smithi102.stdout: "mutes": [] 2024-04-25T07:10:57.479 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.479 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_first_committed": 1, 2024-04-25T07:10:57.479 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_last_committed": 1, 2024-04-25T07:10:57.479 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap": { 2024-04-25T07:10:57.479 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:10:57.479 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:10:57.479 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:07:22.126495Z", 2024-04-25T07:10:57.479 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:07:22.126495Z", 2024-04-25T07:10:57.479 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release": 19, 2024-04-25T07:10:57.479 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release_name": "squid", 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "election_strategy": 1, 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "disallowed_leaders: ": "", 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": false, 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "tiebreaker_mon": "", 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_ranks: ": "", 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "features": { 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "persistent": [ 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "kraken", 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "luminous", 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "mimic", 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap-prune", 2024-04-25T07:10:57.480 INFO:tasks.workunit.client.0.smithi102.stdout: "nautilus", 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: "octopus", 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: "pacific", 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: "elector-pinging", 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: "quincy", 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: "reef", 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: "squid" 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: "optional": [] 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: "mons": [ 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: "rank": 0, 2024-04-25T07:10:57.481 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "a", 2024-04-25T07:10:57.482 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:10:57.482 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.482 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.482 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.482 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:10:57.482 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 0 2024-04-25T07:10:57.482 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.482 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.482 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.482 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:10:57.482 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 0, 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 0, 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_location": "{}" 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: "quorum": [ 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: 0 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap": { 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 41, 2024-04-25T07:10:57.483 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:07:23.218972+0000", 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:10:19.951651+0000", 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "last_up_change": "2024-04-25T07:10:17.528429+0000", 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "last_in_change": "2024-04-25T07:10:19.486446+0000", 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_num": 5800384, 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_set": [ 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "nodown", 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "noin", 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "noout", 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "pglog_hardlimit", 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snapdirs", 2024-04-25T07:10:57.484 INFO:tasks.workunit.client.0.smithi102.stdout: "recovery_deletes", 2024-04-25T07:10:57.485 INFO:tasks.workunit.client.0.smithi102.stdout: "sortbitwise" 2024-04-25T07:10:57.485 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.485 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_version": 9, 2024-04-25T07:10:57.485 INFO:tasks.workunit.client.0.smithi102.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:10:57.485 INFO:tasks.workunit.client.0.smithi102.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:10:57.485 INFO:tasks.workunit.client.0.smithi102.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:10:57.485 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_snapshot": "", 2024-04-25T07:10:57.485 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_max": 1, 2024-04-25T07:10:57.485 INFO:tasks.workunit.client.0.smithi102.stdout: "max_osd": 3, 2024-04-25T07:10:57.485 INFO:tasks.workunit.client.0.smithi102.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:10:57.485 INFO:tasks.workunit.client.0.smithi102.stdout: "min_compat_client": "jewel", 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: "require_osd_release": "squid", 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: "allow_crimson": false, 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: "pools": [ 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_name": "test", 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: "create_time": "2024-04-25T07:09:16.379687+0000", 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": 1, 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_names": "hashpspool", 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 3, 2024-04-25T07:10:57.486 INFO:tasks.workunit.client.0.smithi102.stdout: "min_size": 2, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_rule": 0, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "object_hash": 2, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num": 1, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num": 1, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num_target": 1, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_target": 1, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_pending": 1, 2024-04-25T07:10:57.487 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pg_merge_meta": { 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: "source_pgid": "0.0", 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: "ready_epoch": 0, 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_started": 0, 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": 0, 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: "source_version": "0'0", 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: "target_version": "0'0" 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: "last_change": "29", 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend": "0", 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:10:57.488 INFO:tasks.workunit.client.0.smithi102.stdout: "auid": 0, 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_seq": 0, 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_epoch": 0, 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_snaps": [], 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps": "[]", 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_bytes": 0, 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_objects": 0, 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "tiers": [], 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "tier_of": -1, 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "read_tier": -1, 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "write_tier": -1, 2024-04-25T07:10:57.489 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_mode": "none", 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_bytes": 0, 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_objects": 0, 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_flush_age": 0, 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_evict_age": 0, 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profile": "", 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_params": { 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "none" 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.490 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_period": 0, 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_count": 0, 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "use_gmt_hitset": true, 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "grade_table": [], 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "stripe_width": 0, 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_num_objects": 0, 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "fast_read": false, 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "options": {}, 2024-04-25T07:10:57.491 INFO:tasks.workunit.client.0.smithi102.stdout: "application_metadata": {}, 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: "read_balance": { 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: "score_type": "Fair distribution", 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: "score_acting": 2.9900000095367432, 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: "score_stable": 2.9900000095367432, 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_acting": 2, 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_stable": 2, 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity": 1, 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.492 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.493 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.493 INFO:tasks.workunit.client.0.smithi102.stdout: "osds": [ 2024-04-25T07:10:57.493 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.493 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:10:57.493 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "0936f745-b82d-4ca4-9a42-d371e0fa2286", 2024-04-25T07:10:57.493 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:10:57.493 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:10:57.493 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:10:57.493 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:10:57.493 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 5, 2024-04-25T07:10:57.493 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 29, 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 36, 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 40, 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 30, 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1254124613 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.494 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1254124613 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:10:57.495 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1254124613 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1254124613 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.496 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1254124613 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1254124613 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:10:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1254124613 2024-04-25T07:10:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:10:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1254124613 2024-04-25T07:10:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6803/1254124613", 2024-04-25T07:10:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6805/1254124613", 2024-04-25T07:10:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1254124613", 2024-04-25T07:10:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1254124613", 2024-04-25T07:10:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:10:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:10:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:10:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "056e4b63-b68a-492b-83e9-1ebaac50e9c4", 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 0, 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 0, 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 0, 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 10, 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 23, 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 31, 2024-04-25T07:10:57.500 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3876753015 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3876753015 2024-04-25T07:10:57.501 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3876753015 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.502 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:10:57.503 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3876753015 2024-04-25T07:10:57.503 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.503 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.503 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.503 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:10:57.503 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.503 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.503 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.503 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:10:57.503 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3876753015 2024-04-25T07:10:57.503 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3876753015 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.504 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3876753015 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3876753015 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/3876753015", 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/3876753015", 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3876753015", 2024-04-25T07:10:57.505 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3876753015", 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: "exists" 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "a15e78f2-04f0-4690-aca5-f26e6ea79b64", 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:10:57.506 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 15, 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 29, 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 38, 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 0, 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 32, 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 919078068 2024-04-25T07:10:57.507 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 919078068 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:10:57.508 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 919078068 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 919078068 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 919078068 2024-04-25T07:10:57.509 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 919078068 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:10:57.510 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 919078068 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 919078068 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/919078068", 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/919078068", 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/919078068", 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/919078068", 2024-04-25T07:10:57.511 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_xinfo": [ 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:10:05.067169+0000", 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:10:57.512 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:07:58.290765+0000", 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:10:05.628457+0000", 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:08:34.951458+0000", 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.513 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:10:06.202778+0000", 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:09:11.890666+0000", 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap": [], 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_items": [], 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_primaries": [], 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_temp": [], 2024-04-25T07:10:57.514 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_temp": [], 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: "blocklist": {}, 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: "range_blocklist": {}, 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profiles": { 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: "default": { 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: "k": "2", 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: "m": "2", 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: "plugin": "jerasure", 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: "technique": "reed_sol_van" 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps_queue": [], 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: "new_removed_snaps": [], 2024-04-25T07:10:57.515 INFO:tasks.workunit.client.0.smithi102.stdout: "new_purged_snaps": [], 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_node_flags": {}, 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: "device_class_flags": {}, 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": { 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_enabled": false, 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_bucket_count": 0, 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_bucket": 0 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_metadata": [ 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:10:57.516 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6804/1254124613,v1:127.0.0.1:6805/1254124613]", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:10:57.517 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:07:29.895643Z", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:10:57.518 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6802/1254124613,v1:127.0.0.1:6803/1254124613]", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1254124613,v1:127.0.0.1:6809/1254124613]", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1254124613,v1:127.0.0.1:6807/1254124613]", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:10:57.519 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:10:57.520 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/3876753015,v1:127.0.0.1:6813/3876753015]", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:10:57.521 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:08:06.168581Z", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:10:57.522 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/3876753015,v1:127.0.0.1:6811/3876753015]", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3876753015,v1:127.0.0.1:6817/3876753015]", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3876753015,v1:127.0.0.1:6815/3876753015]", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:10:57.523 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/919078068,v1:127.0.0.1:6813/919078068]", 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:10:57.524 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:10:57.525 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:08:42.938863Z", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:10:57.526 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/919078068,v1:127.0.0.1:6811/919078068]", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/919078068,v1:127.0.0.1:6817/919078068]", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/919078068,v1:127.0.0.1:6815/919078068]", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:10:57.527 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_clean_epochs": { 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": { 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: "per_pool": [] 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_epochs": [ 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 36 2024-04-25T07:10:57.528 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 38 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_first_committed": 1, 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_last_committed": 41, 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: "crushmap": { 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": [ 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.0", 2024-04-25T07:10:57.529 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.1", 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.2", 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.530 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: "types": [ 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 0, 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd" 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "host" 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 2, 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "chassis" 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.531 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 3, 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "rack" 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 4, 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "row" 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 5, 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pdu" 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 6, 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pod" 2024-04-25T07:10:57.532 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 7, 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "room" 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 8, 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "datacenter" 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 9, 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "zone" 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.533 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 10, 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "region" 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "root" 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: "buckets": [ 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -1, 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default", 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:10:57.534 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -2, 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default~hdd", 2024-04-25T07:10:57.535 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:10:57.537 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:10:57.537 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:10:57.537 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102", 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:10:57.538 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.539 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102~hdd", 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.540 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.541 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: "rules": [ 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_id": 0, 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_name": "replicated_rule", 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: "steps": [ 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "take", 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: "item": -1, 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: "item_name": "default" 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "choose_firstn", 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 0, 2024-04-25T07:10:57.542 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "osd" 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "emit" 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: "tunables": { 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_tries": 0, 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_total_tries": 50, 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_stable": 1, 2024-04-25T07:10:57.543 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_descents": 100, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_collision_tries": 100, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "straw_calc_version": 1, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "profile": "jewel", 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_tunables": 1, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "legacy_tunables": 0, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "minimum_required_version": "jewel", 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables": 1, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables2": 1, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v2_rules": 0, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables3": 1, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v3_rules": 0, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v4_buckets": 1, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables5": 1, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v5_rules": 0, 2024-04-25T07:10:57.544 INFO:tasks.workunit.client.0.smithi102.stdout: "has_msr_rules": 0 2024-04-25T07:10:57.545 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.545 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_args": {} 2024-04-25T07:10:57.545 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.545 INFO:tasks.workunit.client.0.smithi102.stdout: "fsmap": { 2024-04-25T07:10:57.545 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:10:57.545 INFO:tasks.workunit.client.0.smithi102.stdout: "default_fscid": -1, 2024-04-25T07:10:57.545 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": { 2024-04-25T07:10:57.545 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": {}, 2024-04-25T07:10:57.545 INFO:tasks.workunit.client.0.smithi102.stdout: "ro_compat": {}, 2024-04-25T07:10:57.545 INFO:tasks.workunit.client.0.smithi102.stdout: "incompat": { 2024-04-25T07:10:57.546 INFO:tasks.workunit.client.0.smithi102.stderr:report 375671496 2024-04-25T07:10:57.546 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_1": "base v0.20", 2024-04-25T07:10:57.546 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:10:57.546 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:10:57.546 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:10:57.546 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:10:57.546 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:10:57.546 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_8": "no anchor table", 2024-04-25T07:10:57.546 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_9": "file layout v2", 2024-04-25T07:10:57.546 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:10:57.546 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_11": "minor log segments", 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_flags": { 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "enable_multiple": true, 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "ever_enabled_multiple": true 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "standbys": [], 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "filesystems": [] 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "auth": { 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 5, 2024-04-25T07:10:57.547 INFO:tasks.workunit.client.0.smithi102.stdout: "num_secrets": 3 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 1, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_active": 1, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_unknown": 0, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osd": 3, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_sum": { 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 61700, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 100, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 300, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 100, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:10:57.548 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 100, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 100, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 100, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:10:57.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:10:57.550 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 0, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 0, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 16, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 16, 2024-04-25T07:10:57.551 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 2, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 2, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 0 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum": { 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 3, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83628, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2112, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:10:57.552 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314489172, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036912128, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2162688, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 460116, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10395, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416933 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:10:57.553 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 1, 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.554 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum_by_class": { 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "hdd": { 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 3, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83628, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2112, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314489172, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:10:57.555 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036912128, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2162688, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 460116, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10395, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416933 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 1, 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:10:57.556 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_stats": [ 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T07:10:57.557 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 1, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 61700, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 100, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 300, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 100, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 100, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:10:57.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 100, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 100, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:10:57.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:10:57.560 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 1228800, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 185100, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 16, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 16, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 2, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 2, 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 3 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.561 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_stats": [ 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 154618822667 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 42949672992 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 163208757258 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.562 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_state": [ 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "active+undersized+degraded", 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 1 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_osd": [ 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:10:57.563 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 1, 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 0, 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": [ 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": {} 2024-04-25T07:10:57.564 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "servicemap": { 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 5, 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:10:08.520163+0000", 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "services": {} 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_last_committed": 96, 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_first_committed": 1, 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_last_committed": 392, 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "paxos": { 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 536, 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pn": 0, 2024-04-25T07:10:57.565 INFO:tasks.workunit.client.0.smithi102.stdout: "accepted_pn": 0 2024-04-25T07:10:57.566 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:10:57.566 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:10:57.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:10:57.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-04-25T07:10:57.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-04-25T07:10:57.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-04-25T07:10:57.567 INFO:tasks.workunit.client.0.smithi102.stdout:writing non-divergent object existing_4 2024-04-25T07:10:57.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-04-25T07:10:57.879 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:10:57.880 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:10:20.429709+0000 29'100 41:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:10:57.880 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:10:57.880 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:10:57.880 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:10:57.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:10:58.011 INFO:tasks.workunit.client.0.smithi102.stdout:delay recovery 2024-04-25T07:10:58.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-04-25T07:10:58.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-04-25T07:10:58.420 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:10:58.420 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:10:20.429709+0000 29'100 41:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:10:58.420 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:10:58.420 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:10:58.420 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:10:58.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:58.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-04-25T07:10:58.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:10:58.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:10:58.437 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:10:58.437 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:58.437 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:10:58.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.0.asok 2024-04-25T07:10:58.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-04-25T07:10:58.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:10:58.552 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:10:58.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:58.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-04-25T07:10:58.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:10:58.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:10:58.566 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:10:58.566 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:58.566 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:10:58.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.2.asok 2024-04-25T07:10:58.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-04-25T07:10:58.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:10:58.677 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:10:58.690 INFO:tasks.workunit.client.0.smithi102.stdout:revive divergent 1 2024-04-25T07:10:58.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-04-25T07:10:58.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-04-25T07:10:59.080 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:10:59.080 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:10:20.429709+0000 41'101 41:127 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:06:55.975828+0000 0 0 2024-04-25T07:10:59.080 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:10:59.080 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:10:59.081 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:10:59.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-04-25T07:10:59.702 INFO:tasks.workunit.client.0.smithi102.stderr:noup is set 2024-04-25T07:10:59.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-04-25T07:10:59.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:10:59.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:10:59.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:10:59.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:10:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:10:59.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:10:59.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:10:59.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:10:59.725 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:10:59.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:10:59.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:10:59.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:10:59.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:10:59.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:10:59.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:10:59.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:10:59.764 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:59.762+0000 7f1b5040a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:59.769 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:59.766+0000 7f1b5040a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:59.771 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:10:59.770+0000 7f1b5040a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:00.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:11:00.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-04-25T07:11:00.308 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:00.306+0000 7f1b5040a7c0 -1 Falling back to public interface 2024-04-25T07:11:03.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:03.422+0000 7f1b5040a7c0 -1 osd.1 29 log_to_monitors true 2024-04-25T07:11:05.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-04-25T07:11:05.194 INFO:tasks.workunit.client.0.smithi102.stdout:delay recovery divergent 2024-04-25T07:11:05.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:05.618 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:05.618 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:10:20.429709+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:06:55.975828+0000 0 0 2024-04-25T07:11:05.618 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:11:05.618 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:05.619 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:11:05.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-04-25T07:11:05.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:11:05.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:11:05.636 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:11:05.636 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:05.636 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:11:05.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.1.asok 2024-04-25T07:11:05.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-04-25T07:11:05.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:11:05.759 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:11:05.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-04-25T07:11:06.511 INFO:tasks.workunit.client.0.smithi102.stderr:noup is unset 2024-04-25T07:11:06.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-04-25T07:11:06.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:11:06.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:11:06.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:11:06.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:11:06.533 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:11:06.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:06.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:11:06.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:11:06.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:07.077 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/1254124613,v1:127.0.0.1:6803/1254124613] [v2:127.0.0.1:6804/1254124613,v1:127.0.0.1:6805/1254124613] exists,up 0936f745-b82d-4ca4-9a42-d371e0fa2286 2024-04-25T07:11:07.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:11:07.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:11:07.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:11:07.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-04-25T07:11:07.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:11:07.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:11:07.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:11:07.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:11:07.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:07.079 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:11:07.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:11:07.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:07.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:07.558 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1927207044,v1:127.0.0.1:6819/1927207044] [v2:127.0.0.1:6820/1927207044,v1:127.0.0.1:6821/1927207044] exists,up 056e4b63-b68a-492b-83e9-1ebaac50e9c4 2024-04-25T07:11:07.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:11:07.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:11:07.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:11:07.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-04-25T07:11:07.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:11:07.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:11:07.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:11:07.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:11:07.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:07.560 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:11:07.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:11:07.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:07.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:08.002 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/919078068,v1:127.0.0.1:6811/919078068] [v2:127.0.0.1:6812/919078068,v1:127.0.0.1:6813/919078068] exists,up a15e78f2-04f0-4690-aca5-f26e6ea79b64 2024-04-25T07:11:08.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:11:08.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:11:08.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:11:08.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:08.388 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:08.388 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:10:20.429709+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:06:55.975828+0000 0 0 2024-04-25T07:11:08.388 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:11:08.388 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:08.389 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:11:08.403 INFO:tasks.workunit.client.0.smithi102.stdout:wait for peering 2024-04-25T07:11:08.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-04-25T07:11:08.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:08.801 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:08.801 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:10:20.429709+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:06:55.975828+0000 0 0 2024-04-25T07:11:08.801 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:11:08.801 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:08.802 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:11:08.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-04-25T07:11:08.973 INFO:tasks.workunit.client.0.smithi102.stdout:killing divergent 1 2024-04-25T07:11:08.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-04-25T07:11:08.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:09.410 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:09.410 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:10:20.429709+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:06:55.975828+0000 0 0 2024-04-25T07:11:09.410 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:11:09.410 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:09.410 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:11:09.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:11:09.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:11:09.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:11:09.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:11:09.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:11:09.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:11:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:11:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-04-25T07:11:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:09.534 INFO:tasks.workunit.client.0.smithi102.stdout:reviving divergent 1 2024-04-25T07:11:09.912 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:09.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:10:20.429709+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:06:55.975828+0000 0 0 2024-04-25T07:11:09.913 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:11:09.913 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:09.913 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:11:09.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-04-25T07:11:09.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:11:09.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:11:09.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:11:09.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:11:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:11:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:11:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:11:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:11:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:11:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:11:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:11:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:11:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:11:09.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:11:09.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:11:09.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:11:09.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:11:09.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:11:09.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:11:09.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:11:09.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:11:09.932 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:11:09.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:11:09.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:11:09.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:11:09.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:11:09.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:11:09.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:11:09.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:11:09.978 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:09.978+0000 7fd25dd867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:09.980 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:09.978+0000 7fd25dd867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:09.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:09.978+0000 7fd25dd867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:10.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:11:10.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:11:10.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:11:10.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:11:10.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:11:10.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:10.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:11:10.376 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:11:10.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:10.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:10.535 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:10.534+0000 7fd25dd867c0 -1 Falling back to public interface 2024-04-25T07:11:10.820 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/1927207044,v1:127.0.0.1:6819/1927207044] [v2:127.0.0.1:6820/1927207044,v1:127.0.0.1:6821/1927207044] exists,up 056e4b63-b68a-492b-83e9-1ebaac50e9c4 2024-04-25T07:11:10.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:11:10.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:11:10.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:11:10.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-04-25T07:11:13.614 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:13.614+0000 7fd25dd867c0 -1 osd.1 47 log_to_monitors true 2024-04-25T07:11:30.822 INFO:tasks.workunit.client.0.smithi102.stdout:allowing recovery 2024-04-25T07:11:30.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-04-25T07:11:30.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:31.219 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:31.219 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-04-25T07:11:15.381051+0000 47'102 52:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:25:02.728234+0000 0 0 2024-04-25T07:11:31.219 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:11:31.219 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:31.219 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:11:31.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T07:11:31.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:11:31.387 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:11:31.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T07:11:31.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:11:31.547 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:11:31.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T07:11:31.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:11:31.707 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:11:31.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-04-25T07:11:31.722 INFO:tasks.workunit.client.0.smithi102.stdout:reading divergent objects 2024-04-25T07:11:31.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:32.106 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:32.106 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-04-25T07:11:15.381051+0000 47'102 52:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:16.435891+0000 0'0 2024-04-25T07:09:16.435891+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:25:02.728234+0000 0 0 2024-04-25T07:11:32.106 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:11:32.106 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:32.106 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:11:32.122 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-04-25T07:11:32.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-04-25T07:11:32.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:11:32.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:11:32.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:11:32.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:11:32.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:11:32.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:11:32.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:11:32.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:11:32.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-04-25T07:11:32.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-04-25T07:11:32.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:11:32.321 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:06.954+0000 7f1b34394640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-04-25T07:09:53.483585+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:11:32.321 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:06.954+0000 7f1b34394640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-04-25T07:09:53.483585+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:11:32.322 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:06.954+0000 7f1b34394640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:11:32.322 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:06.954+0000 7f1b34394640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-04-25T07:11:32.322 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:06.954+0000 7f1b34394640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-04-25T07:11:32.322 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:06.954+0000 7f1b34394640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:11:32.322 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:06.954+0000 7f1b34394640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-04-25T07:11:32.322 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:06.954+0000 7f1b34394640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-04-25T07:09:53.483585+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:11:32.322 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:06.954+0000 7f1b34394640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-04-25T07:11:32.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-04-25T07:11:32.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:11:32.327 INFO:tasks.workunit.client.0.smithi102.stdout:success 2024-04-25T07:11:32.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-04-25T07:11:32.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-04-25T07:11:32.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:11:32.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:11:32.860 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T07:11:32.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-04-25T07:11:32.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:11:32.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:11:32.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:11:32.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:11:32.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:11:38.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:11:38.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:11:38.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:11:38.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:11:38.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:11:38.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:11:38.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:11:38.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:11:38.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:11:38.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:11:38.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:11:38.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:11:38.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:11:38.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:11:38.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:11:38.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:11:38.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:11:38.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:11:38.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:11:38.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:11:38.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:11:38.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:11:38.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:11:38.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:11:38.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:11:38.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:38.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:11:38.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26156 2024-04-25T07:11:38.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:11:38.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:11:38.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:11:38.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:11:38.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:11:38.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:11:38.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:11:38.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:11:38.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:11:38.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:11:38.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:11:38.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:11:38.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:11:38.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:11:38.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:11:38.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:11:38.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:11:38.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:38.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:11:38.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26156 2024-04-25T07:11:38.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:11:38.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:11:38.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:11:38.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:11:38.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:38.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:11:38.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26156 2024-04-25T07:11:38.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:11:38.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:11:38.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:11:38.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:11:38.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-04-25T07:11:38.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-04-25T07:11:38.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-04-25T07:11:38.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-04-25T07:11:38.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-04-25T07:11:38.252 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-04-25T07:11:38.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-04-25T07:11:38.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-04-25T07:11:38.253 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:11:38.253 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:11:38.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-04-25T07:11:38.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:11:38.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:11:38.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:11:38.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:11:38.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:11:38.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:11:38.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:11:38.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:11:38.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:11:38.751 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:11:38.751 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:38.752 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:11:38.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:11:38.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:11:38.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:11:38.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:11:38.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:11:38.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:11:38.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:11:38.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:11:38.973 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:11:38.973 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:11:38.973 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:11:38.974 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:11:38.975 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:38.975 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:11:38.975 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-mon.a.asok 2024-04-25T07:11:38.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:11:38.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26156/ceph-mon.a.asok config get fsid 2024-04-25T07:11:39.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:11:39.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:11:39.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:11:39.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:11:39.092 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:11:39.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:11:39.092 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:11:39.092 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:11:39.092 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:11:39.092 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:39.092 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:11:39.093 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-mon.a.asok 2024-04-25T07:11:39.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:11:39.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26156/ceph-mon.a.asok config get mon_host 2024-04-25T07:11:39.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-04-25T07:11:39.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:11:39.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:11:39.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:11:39.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:11:39.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:11:39.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:11:40.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:11:40.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:11:40.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:11:40.125 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:11:40.125 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:40.125 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:11:40.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:11:40.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:11:40.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:11:40.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:11:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:11:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:11:40.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:11:40.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:11:40.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:11:40.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:11:40.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:11:40.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:11:40.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0e4773f-42a0-4d5c-b371-232090487d33 2024-04-25T07:11:40.169 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 b0e4773f-42a0-4d5c-b371-232090487d33 2024-04-25T07:11:40.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b0e4773f-42a0-4d5c-b371-232090487d33' 2024-04-25T07:11:40.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:11:40.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAsAipmw9V4CxAAv/0wVRnCiUK//sx4PUvKkQ== 2024-04-25T07:11:40.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAsAipmw9V4CxAAv/0wVRnCiUK//sx4PUvKkQ=="}' 2024-04-25T07:11:40.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0e4773f-42a0-4d5c-b371-232090487d33 -i td/divergent-priors/0/new.json 2024-04-25T07:11:40.524 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:11:40.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:11:40.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsAipmw9V4CxAAv/0wVRnCiUK//sx4PUvKkQ== --osd-uuid b0e4773f-42a0-4d5c-b371-232090487d33 2024-04-25T07:11:40.584 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:40.581+0000 7f0c2d7487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:40.600 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:40.597+0000 7f0c2d7487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:40.602 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:40.601+0000 7f0c2d7487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:40.602 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:40.601+0000 7f0c2d7487c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:11:40.602 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:40.601+0000 7f0c2d7487c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:11:40.602 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:40.601+0000 7f0c2d7487c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:11:40.632 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:40.629+0000 7f0c2d7487c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:11:46.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:11:46.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:11:46.941 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:11:46.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:11:46.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:11:47.536 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:11:47.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:11:47.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:11:47.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:11:47.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:11:47.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:11:47.575 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:47.573+0000 7f49d520a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:47.599 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:47.597+0000 7f49d520a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:47.600 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:47.597+0000 7f49d520a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:47.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:11:47.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:11:47.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:11:47.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:11:47.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:11:47.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:47.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:11:47.979 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:11:47.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:47.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:11:48.171 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:48.169+0000 7f49d520a7c0 -1 Falling back to public interface 2024-04-25T07:11:48.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:49.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:49.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:49.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:11:49.389 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:11:49.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:49.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:11:49.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:50.723 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:11:50.721+0000 7f49d520a7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:11:50.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:50.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:50.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:11:50.855 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:11:50.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:50.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:11:51.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:52.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:52.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:52.455 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:11:52.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:11:52.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:52.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:11:53.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:54.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:54.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:54.005 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:11:54.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:11:54.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:54.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:11:54.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:55.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:55.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:55.488 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:11:55.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:11:55.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:55.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:11:55.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:56.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:56.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:56.958 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:11:56.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:11:56.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:56.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:11:57.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:58.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:58.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:58.564 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:11:58.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:11:58.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:58.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:11:58.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:11:59.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:59.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:59.977 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:11:59.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:11:59.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:59.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:00.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:01.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:01.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:01.647 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:12:01.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:12:01.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:01.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:02.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:03.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:03.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:03.090 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:12:03.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:12:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:03.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:04.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:04.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:04.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:12:04.641 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:12:04.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:04.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:05.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:06.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:06.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:06.627 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:12:06.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:12:06.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:06.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:07.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:08.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:08.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:08.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:12:08.309 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:12:08.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:08.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:09.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:10.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:10.038 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:12:10.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:10.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:12:10.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:10.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:10.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:11.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:11.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:11.829 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:12:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:12:11.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:11.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:12.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:13.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:13.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:13.811 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:12:13.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:12:13.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:13.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:14.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:15.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:15.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:15.380 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:12:15.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:12:15.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:15.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:15.824 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/746209809,v1:127.0.0.1:6803/746209809] [v2:127.0.0.1:6804/746209809,v1:127.0.0.1:6805/746209809] exists,up b0e4773f-42a0-4d5c-b371-232090487d33 2024-04-25T07:12:15.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:12:15.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:12:15.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:12:15.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:15.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:12:15.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:12:15.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:12:15.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:12:15.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:15.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:15.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:15.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:15.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:12:15.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:12:15.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:12:15.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:12:15.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0fa44109-b9ae-4d0d-9eb0-a7b7d85f02fe 2024-04-25T07:12:15.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0fa44109-b9ae-4d0d-9eb0-a7b7d85f02fe' 2024-04-25T07:12:15.832 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 0fa44109-b9ae-4d0d-9eb0-a7b7d85f02fe 2024-04-25T07:12:15.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:12:15.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPAipms2vcMhAAlxlzd1n2+iAWyO4mCxNVuQ== 2024-04-25T07:12:15.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPAipms2vcMhAAlxlzd1n2+iAWyO4mCxNVuQ=="}' 2024-04-25T07:12:15.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0fa44109-b9ae-4d0d-9eb0-a7b7d85f02fe -i td/divergent-priors/1/new.json 2024-04-25T07:12:16.362 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:12:16.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:12:16.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPAipms2vcMhAAlxlzd1n2+iAWyO4mCxNVuQ== --osd-uuid 0fa44109-b9ae-4d0d-9eb0-a7b7d85f02fe 2024-04-25T07:12:16.425 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:16.425+0000 7f4c899a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:16.427 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:16.425+0000 7f4c899a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:16.429 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:16.425+0000 7f4c899a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:16.429 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:16.429+0000 7f4c899a37c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:12:16.429 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:16.429+0000 7f4c899a37c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:12:16.429 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:16.429+0000 7f4c899a37c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:12:16.459 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:16.457+0000 7f4c899a37c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:12:23.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:12:23.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:12:23.194 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:12:23.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:12:23.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:12:23.772 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:12:23.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:12:23.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:12:23.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:12:23.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:12:23.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:12:23.806 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:23.805+0000 7f64057457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:23.834 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:23.833+0000 7f64057457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:23.836 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:23.833+0000 7f64057457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:24.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:12:24.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:12:24.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:12:24.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:12:24.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:12:24.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:24.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:12:24.197 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:12:24.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:24.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:24.383 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:24.381+0000 7f64057457c0 -1 Falling back to public interface 2024-04-25T07:12:24.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:25.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:25.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:25.613 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:12:25.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:12:25.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:25.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:26.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:27.016 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:27.013+0000 7f64057457c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:12:27.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:27.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:27.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:12:27.035 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:12:27.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:27.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:27.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:28.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:28.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:28.779 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:12:28.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:12:28.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:28.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:29.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:30.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:30.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:30.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:12:30.319 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:12:30.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:30.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:30.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:31.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:31.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:31.884 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:12:31.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:12:31.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:31.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:32.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:33.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:33.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:33.322 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:12:33.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:12:33.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:33.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:33.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:34.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:34.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:34.940 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:12:34.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:12:34.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:34.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:35.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:36.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:36.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:36.529 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:12:36.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:12:36.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:36.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:37.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:38.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:38.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:38.135 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:12:38.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:12:38.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:38.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:38.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:39.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:39.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:39.577 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:12:39.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:12:39.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:39.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:40.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:41.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:41.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:41.060 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:12:41.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:12:41.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:41.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:41.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:42.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:42.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:42.669 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:12:42.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:12:42.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:42.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:43.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:44.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:44.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:44.307 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:12:44.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:12:44.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:44.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:45.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:46.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:46.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:46.086 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:12:46.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:12:46.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:46.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:46.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:47.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:47.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:47.840 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:12:47.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:12:47.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:47.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:48.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:49.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:49.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:49.576 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:12:49.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:12:49.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:49.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:50.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:51.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:51.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:51.688 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:12:51.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:12:51.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:51.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:12:52.146 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2763816297,v1:127.0.0.1:6811/2763816297] [v2:127.0.0.1:6812/2763816297,v1:127.0.0.1:6813/2763816297] exists,up 0fa44109-b9ae-4d0d-9eb0-a7b7d85f02fe 2024-04-25T07:12:52.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:12:52.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:12:52.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:12:52.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:12:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:12:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:12:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:12:52.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:12:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f0ddadd5-e679-4f1d-9dff-ccebbee481e3 2024-04-25T07:12:52.152 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 f0ddadd5-e679-4f1d-9dff-ccebbee481e3 2024-04-25T07:12:52.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f0ddadd5-e679-4f1d-9dff-ccebbee481e3' 2024-04-25T07:12:52.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:12:52.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB0AipmHmZ6ChAAJorLj6kNBl/KHUC5BfWKoQ== 2024-04-25T07:12:52.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB0AipmHmZ6ChAAJorLj6kNBl/KHUC5BfWKoQ=="}' 2024-04-25T07:12:52.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f0ddadd5-e679-4f1d-9dff-ccebbee481e3 -i td/divergent-priors/2/new.json 2024-04-25T07:12:52.699 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:12:52.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:12:52.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB0AipmHmZ6ChAAJorLj6kNBl/KHUC5BfWKoQ== --osd-uuid f0ddadd5-e679-4f1d-9dff-ccebbee481e3 2024-04-25T07:12:52.757 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:52.757+0000 7fd37dbea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:52.772 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:52.773+0000 7fd37dbea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:52.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:52.773+0000 7fd37dbea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:52.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:52.773+0000 7fd37dbea7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:12:52.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:52.773+0000 7fd37dbea7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:12:52.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:52.773+0000 7fd37dbea7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:12:52.845 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:12:52.845+0000 7fd37dbea7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:12:59.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:12:59.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:12:59.727 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:12:59.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:12:59.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:13:00.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:13:00.296 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:13:00.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:13:00.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:13:00.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:13:00.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:13:00.329 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:13:00.329+0000 7f7f814797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:00.346 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:13:00.345+0000 7f7f814797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:00.347 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:13:00.345+0000 7f7f814797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:00.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:13:00.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:13:00.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:13:00.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:13:00.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:13:00.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:00.722 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:13:00.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:13:00.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:00.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:00.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:13:00.901+0000 7f7f814797c0 -1 Falling back to public interface 2024-04-25T07:13:01.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:02.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:02.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:02.140 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:13:02.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:13:02.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:02.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:02.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:03.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:03.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:03.572 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:13:03.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:13:03.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:03.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:03.630 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:13:03.629+0000 7f7f814797c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:13:04.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:05.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:05.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:05.136 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:13:05.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:13:05.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:05.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:05.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:06.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:06.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:06.579 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:13:06.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:13:06.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:06.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:07.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:08.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:08.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:08.082 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:13:08.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:13:08.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:08.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:08.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:09.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:09.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:09.575 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:13:09.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:13:09.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:09.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:10.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:11.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:11.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:11.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:13:11.021 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:13:11.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:11.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:11.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:12.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:12.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:12.581 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:13:12.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:13:12.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:12.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:13.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:14.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:14.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:14.045 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:13:14.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:13:14.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:14.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:14.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:15.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:15.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:15.632 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:13:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:13:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:16.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:17.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:17.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:17.116 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:13:17.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:13:17.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:17.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:17.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:18.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:18.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:18.913 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:13:18.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:13:18.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:18.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:19.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:20.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:20.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:20.929 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:13:20.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:13:20.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:20.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:22.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:23.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:23.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:23.042 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:13:23.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:13:23.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:23.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:23.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:24.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:24.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:24.626 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:13:24.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:13:24.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:24.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:25.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:26.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:26.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:26.195 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:13:26.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:13:26.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:26.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:27.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:28.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:28.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:28.102 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:13:28.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:13:28.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:28.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:28.543 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2494085943,v1:127.0.0.1:6819/2494085943] [v2:127.0.0.1:6820/2494085943,v1:127.0.0.1:6821/2494085943] exists,up f0ddadd5-e679-4f1d-9dff-ccebbee481e3 2024-04-25T07:13:28.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:13:28.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:13:28.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:13:28.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-04-25T07:13:29.374 INFO:tasks.workunit.client.0.smithi102.stderr:noout is set 2024-04-25T07:13:29.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-04-25T07:13:30.086 INFO:tasks.workunit.client.0.smithi102.stderr:noin is set 2024-04-25T07:13:30.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-04-25T07:13:31.073 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T07:13:31.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-04-25T07:13:31.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T07:13:31.776 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T07:13:31.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:13:32.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-04-25T07:13:33.577 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 3 2024-04-25T07:13:33.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-04-25T07:13:34.339 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 min_size to 2 2024-04-25T07:13:34.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-04-25T07:13:34.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:13:34.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:13:34.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:13:34.824 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:13:34.824 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:13:34.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:13:34.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:13:34.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:13:34.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836500 2024-04-25T07:13:34.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836500 2024-04-25T07:13:34.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500' 2024-04-25T07:13:34.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:13:34.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:13:35.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672976 2024-04-25T07:13:35.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672976 2024-04-25T07:13:35.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672976' 2024-04-25T07:13:35.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:13:35.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:13:35.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509451 2024-04-25T07:13:35.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509451 2024-04-25T07:13:35.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672976 2-64424509451' 2024-04-25T07:13:35.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:13:35.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836500 2024-04-25T07:13:35.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:13:35.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:13:35.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836500 2024-04-25T07:13:35.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:13:35.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836500 2024-04-25T07:13:35.302 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836500 2024-04-25T07:13:35.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-04-25T07:13:35.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:13:35.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836498 -lt 21474836500 2024-04-25T07:13:35.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:13:36.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:13:36.701 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:13:37.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836500 2024-04-25T07:13:37.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:13:37.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672976 2024-04-25T07:13:37.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:13:37.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:13:37.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672976 2024-04-25T07:13:37.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:13:37.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672976 2024-04-25T07:13:37.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-04-25T07:13:37.134 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672976 2024-04-25T07:13:37.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:13:37.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672976 2024-04-25T07:13:37.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:13:37.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509451 2024-04-25T07:13:37.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:13:37.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:13:37.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509451 2024-04-25T07:13:37.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:13:37.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509451 2024-04-25T07:13:37.562 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509451 2024-04-25T07:13:37.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-04-25T07:13:37.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:13:37.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509451 -lt 64424509451 2024-04-25T07:13:37.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-04-25T07:13:37.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:13:37.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:13:37.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:13:37.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:13:37.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:13:37.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:13:37.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:13:37.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:13:37.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:13:38.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:13:38.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:13:38.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:13:38.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:13:38.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:13:38.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:13:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:13:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:13:38.577 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:13:38.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:13:38.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:13:38.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:13:38.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836503 2024-04-25T07:13:38.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836503 2024-04-25T07:13:38.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503' 2024-04-25T07:13:38.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:13:38.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:13:38.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672979 2024-04-25T07:13:38.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672979 2024-04-25T07:13:38.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672979' 2024-04-25T07:13:38.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:13:38.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:13:39.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509454 2024-04-25T07:13:39.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509454 2024-04-25T07:13:39.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672979 2-64424509454' 2024-04-25T07:13:39.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:13:39.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836503 2024-04-25T07:13:39.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:13:39.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:13:39.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836503 2024-04-25T07:13:39.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:13:39.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836503 2024-04-25T07:13:39.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-04-25T07:13:39.043 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836503 2024-04-25T07:13:39.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:13:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836503 2024-04-25T07:13:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:13:39.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672979 2024-04-25T07:13:39.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:13:39.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:13:39.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672979 2024-04-25T07:13:39.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:13:39.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672979 2024-04-25T07:13:39.493 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672979 2024-04-25T07:13:39.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-04-25T07:13:39.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:13:39.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672977 -lt 42949672979 2024-04-25T07:13:39.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:13:40.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:13:40.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:13:41.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672979 2024-04-25T07:13:41.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:13:41.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509454 2024-04-25T07:13:41.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:13:41.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:13:41.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509454 2024-04-25T07:13:41.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:13:41.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509454 2024-04-25T07:13:41.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-04-25T07:13:41.387 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509454 2024-04-25T07:13:41.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:13:41.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509454 2024-04-25T07:13:41.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:13:41.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:13:41.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:13:42.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:13:42.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:13:42.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:13:42.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:13:42.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:13:42.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:13:42.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:13:42.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:13:42.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:13:42.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:13:42.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:13:42.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:13:43.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:13:43.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:13:43.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:13:43.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-04-25T07:13:43.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-04-25T07:13:43.719 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:13:43.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-04-25T07:13:43.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-04-25T07:13:43.735 INFO:tasks.workunit.client.0.smithi102.stdout:primary and soon to be divergent is 1 2024-04-25T07:13:43.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:13:44.118 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:13:44.118 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:13:32.669455+0000 0'0 29:21 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:15:49.970380+0000 0 0 2024-04-25T07:13:44.118 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:13:44.118 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:13:44.118 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:13:44.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-04-25T07:13:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T07:13:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-04-25T07:13:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-04-25T07:13:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T07:13:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-04-25T07:13:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-04-25T07:13:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T07:13:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-04-25T07:13:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-04-25T07:13:44.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-04-25T07:13:44.135 INFO:tasks.workunit.client.0.smithi102.stdout:writing initial objects 2024-04-25T07:13:44.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-04-25T07:13:44.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:44.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-04-25T07:13:44.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:44.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-04-25T07:13:44.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:44.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-04-25T07:13:44.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:44.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-04-25T07:13:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-04-25T07:13:45.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:45.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-04-25T07:13:45.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:45.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-04-25T07:13:45.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:45.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-04-25T07:13:45.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:45.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-04-25T07:13:45.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:45.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-04-25T07:13:45.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:45.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-04-25T07:13:46.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:46.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-04-25T07:13:46.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:46.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-04-25T07:13:46.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:46.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-04-25T07:13:46.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:46.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-04-25T07:13:46.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:46.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-04-25T07:13:46.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:46.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-04-25T07:13:47.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:47.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-04-25T07:13:47.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:47.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-04-25T07:13:47.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:47.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-04-25T07:13:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-04-25T07:13:47.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:47.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-04-25T07:13:48.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:48.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-04-25T07:13:48.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:48.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-04-25T07:13:48.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:48.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-04-25T07:13:48.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:48.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-04-25T07:13:48.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:48.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-04-25T07:13:48.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:48.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-04-25T07:13:49.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:49.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-04-25T07:13:49.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:49.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-04-25T07:13:49.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:49.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-04-25T07:13:49.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:49.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-04-25T07:13:49.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:49.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-04-25T07:13:49.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:49.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-04-25T07:13:50.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:50.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-04-25T07:13:50.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:50.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-04-25T07:13:50.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:50.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-04-25T07:13:50.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:50.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-04-25T07:13:50.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:50.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-04-25T07:13:50.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:50.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-04-25T07:13:51.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:51.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-04-25T07:13:51.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:51.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-04-25T07:13:51.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:51.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-04-25T07:13:51.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:51.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-04-25T07:13:51.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-04-25T07:13:52.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:52.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-04-25T07:13:52.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:52.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-04-25T07:13:52.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:52.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-04-25T07:13:52.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:52.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-04-25T07:13:52.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:52.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-04-25T07:13:52.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:52.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-04-25T07:13:53.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:53.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-04-25T07:13:53.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:53.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-04-25T07:13:53.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:53.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-04-25T07:13:53.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:53.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-04-25T07:13:53.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:53.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-04-25T07:13:53.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:53.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-04-25T07:13:54.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:54.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-04-25T07:13:54.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:54.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-04-25T07:13:54.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:54.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-04-25T07:13:54.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:54.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-04-25T07:13:54.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:54.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-04-25T07:13:55.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:55.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-04-25T07:13:55.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:55.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-04-25T07:13:55.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:55.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-04-25T07:13:55.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:55.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-04-25T07:13:55.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:55.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-04-25T07:13:56.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:56.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-04-25T07:13:56.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:56.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-04-25T07:13:56.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:56.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-04-25T07:13:56.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:56.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-04-25T07:13:56.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:56.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-04-25T07:13:56.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:56.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-04-25T07:13:57.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:57.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-04-25T07:13:57.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:57.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-04-25T07:13:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-04-25T07:13:57.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:57.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-04-25T07:13:57.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:57.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-04-25T07:13:58.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:58.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-04-25T07:13:58.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:58.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-04-25T07:13:58.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:58.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-04-25T07:13:58.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:58.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-04-25T07:13:58.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:58.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-04-25T07:13:58.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:58.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-04-25T07:13:59.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:59.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-04-25T07:13:59.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:59.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-04-25T07:13:59.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:59.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-04-25T07:13:59.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:59.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-04-25T07:13:59.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:59.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-04-25T07:13:59.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:13:59.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-04-25T07:14:00.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:00.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-04-25T07:14:00.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:00.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-04-25T07:14:00.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:00.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-04-25T07:14:00.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:00.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-04-25T07:14:00.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:00.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-04-25T07:14:00.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:00.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-04-25T07:14:01.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:01.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-04-25T07:14:01.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:01.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-04-25T07:14:01.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:01.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-04-25T07:14:01.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:01.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-04-25T07:14:01.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:14:01.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-04-25T07:14:01.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:14:01.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:14:01.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:14:01.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:14:01.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:14:01.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:14:01.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:14:01.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:14:01.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:14:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:14:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:14:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:14:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:14:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:14:01.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:14:02.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:14:02.382 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:14:02.382 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:14:02.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:14:02.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:02.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:14:02.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836510 2024-04-25T07:14:02.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836510 2024-04-25T07:14:02.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510' 2024-04-25T07:14:02.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:02.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:14:02.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672986 2024-04-25T07:14:02.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672986 2024-04-25T07:14:02.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672986' 2024-04-25T07:14:02.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:02.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:14:02.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509460 2024-04-25T07:14:02.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509460 2024-04-25T07:14:02.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672986 2-64424509460' 2024-04-25T07:14:02.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:02.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836510 2024-04-25T07:14:02.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:02.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:14:02.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836510 2024-04-25T07:14:02.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:02.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836510 2024-04-25T07:14:02.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-04-25T07:14:02.889 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836510 2024-04-25T07:14:02.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:14:03.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836510 2024-04-25T07:14:03.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:03.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672986 2024-04-25T07:14:03.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:03.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:14:03.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672986 2024-04-25T07:14:03.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:03.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672986 2024-04-25T07:14:03.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-04-25T07:14:03.323 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672986 2024-04-25T07:14:03.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:14:03.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672986 -lt 42949672986 2024-04-25T07:14:03.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:03.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509460 2024-04-25T07:14:03.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:03.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:14:03.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509460 2024-04-25T07:14:03.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:03.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509460 2024-04-25T07:14:03.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-04-25T07:14:03.747 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509460 2024-04-25T07:14:03.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:14:04.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509460 2024-04-25T07:14:04.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:14:05.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:14:05.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:14:05.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509461 -lt 64424509460 2024-04-25T07:14:05.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:14:05.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:05.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:06.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:14:06.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:14:06.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:14:06.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:14:06.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:14:06.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:14:06.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:14:06.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:14:06.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:14:06.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:14:06.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:06.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:07.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:14:07.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:14:07.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:14:07.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-04-25T07:14:07.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:14:07.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:14:07.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:14:07.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:14:07.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-04-25T07:14:07.574 INFO:tasks.workunit.client.0.smithi102.stdout:blackholing osds 0 2 2024-04-25T07:14:07.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-04-25T07:14:07.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:14:07.971 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:14:07.971 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T07:13:32.669455+0000 29'100 29:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:15:49.970380+0000 0 0 2024-04-25T07:14:07.972 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:14:07.972 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:14:07.972 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:14:07.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:14:07.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-04-25T07:14:07.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:14:07.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:14:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:14:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:14:07.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.0.asok 2024-04-25T07:14:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:14:07.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:14:08.094 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:14:08.095 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:14:08.095 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:14:08.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:14:08.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-04-25T07:14:08.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:14:08.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:14:08.109 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:14:08.109 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:08.109 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:14:08.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.2.asok 2024-04-25T07:14:08.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:14:08.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:14:08.214 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:14:08.214 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:14:08.214 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:14:08.226 INFO:tasks.workunit.client.0.smithi102.stdout:create new divergent objects 2024-04-25T07:14:08.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-04-25T07:14:08.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-04-25T07:14:08.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:14:08.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-04-25T07:14:08.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:14:08.230 INFO:tasks.workunit.client.0.smithi102.stdout:writing divergent objects 2024-04-25T07:14:08.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-04-25T07:14:08.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-04-25T07:14:08.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-04-25T07:14:08.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:14:08.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-04-25T07:14:08.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:14:08.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-04-25T07:14:08.232 INFO:tasks.workunit.client.0.smithi102.stdout:remove divergent objects 2024-04-25T07:14:08.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-04-25T07:14:08.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-04-25T07:14:08.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:14:08.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-04-25T07:14:08.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-04-25T07:14:08.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:14:08.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-04-25T07:14:08.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-04-25T07:14:08.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-04-25T07:14:08.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-04-25T07:14:08.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-04-25T07:14:18.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-04-25T07:14:18.246 INFO:tasks.workunit.client.0.smithi102.stderr:rados: no process found 2024-04-25T07:14:18.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-04-25T07:14:18.246 INFO:tasks.workunit.client.0.smithi102.stdout:killing all the osds 2024-04-25T07:14:18.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:14:18.657 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:14:18.657 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 active+clean 2024-04-25T07:13:32.669455+0000 29'106 29:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:15:49.970380+0000 0 0 2024-04-25T07:14:18.657 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:14:18.657 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:14:18.657 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:14:18.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:14:18.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:14:18.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:14:18.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:14:18.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:14:18.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:14:18.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:14:18.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T07:14:18.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-04-25T07:14:19.310 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T07:14:19.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T07:14:19.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-04-25T07:14:19.905 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already down. 2024-04-25T07:14:19.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T07:14:19.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-04-25T07:14:20.456 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already down. 2024-04-25T07:14:20.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:14:20.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-04-25T07:14:21.029 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. 2024-04-25T07:14:21.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:14:21.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-04-25T07:14:21.857 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already out. 2024-04-25T07:14:21.873 INFO:tasks.workunit.client.0.smithi102.stdout:bringing up non_divergent 0 2 2024-04-25T07:14:21.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-04-25T07:14:21.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:14:22.277 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:14:22.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 stale+active+clean 2024-04-25T07:13:32.669455+0000 29'106 29:127 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:15:49.970380+0000 0 0 2024-04-25T07:14:22.278 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:14:22.278 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:14:22.278 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:14:22.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:14:22.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-04-25T07:14:22.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:14:22.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:14:22.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:14:22.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:14:22.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:14:22.298 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:14:22.298 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:22.298 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:14:22.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:14:22.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:14:22.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:14:22.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:14:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:14:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:14:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:14:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:14:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:14:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:14:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:14:22.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:14:22.300 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:14:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:14:22.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:14:22.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:14:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:14:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:14:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:14:22.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:14:22.332 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:14:22.332+0000 7fd3dfdcf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:22.353 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:14:22.352+0000 7fd3dfdcf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:22.354 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:14:22.352+0000 7fd3dfdcf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:22.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:14:22.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:14:22.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:14:22.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:14:22.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:14:22.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:22.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:14:22.737 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:14:22.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:22.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:22.917 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:14:22.916+0000 7fd3dfdcf7c0 -1 Falling back to public interface 2024-04-25T07:14:23.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:24.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:24.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:24.183 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:14:24.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:14:24.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:24.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:24.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:25.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:25.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:25.605 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:14:25.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:14:25.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:25.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:26.010 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:14:26.008+0000 7fd3dfdcf7c0 -1 osd.0 29 log_to_monitors true 2024-04-25T07:14:26.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:27.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:27.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:27.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:14:27.048 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:14:27.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:27.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:14:27.509 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/2016169313,v1:127.0.0.1:6803/2016169313] [v2:127.0.0.1:6804/2016169313,v1:127.0.0.1:6805/2016169313] exists,up b0e4773f-42a0-4d5c-b371-232090487d33 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:14:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:14:27.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:14:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:14:27.516 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:14:27.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:14:27.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:14:27.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:14:27.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:14:27.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:14:27.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:14:27.564 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:14:27.564+0000 7f02add517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:27.572 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:14:27.572+0000 7f02add517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:27.574 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:14:27.572+0000 7f02add517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:14:27.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:14:27.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:14:27.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:14:27.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:14:27.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:14:27.952 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:14:27.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:14:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:14:28.130 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:14:28.128+0000 7f02add517c0 -1 Falling back to public interface 2024-04-25T07:14:28.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:29.391 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:14:29.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:14:29.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:29.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:14:29.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:30.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:30.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:30.833 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:14:30.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:14:30.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:30.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:14:31.184 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:14:31.184+0000 7f02add517c0 -1 osd.2 29 log_to_monitors true 2024-04-25T07:14:31.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:32.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:32.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:32.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:14:32.266 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:14:32.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:32.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:14:32.693 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/136562268,v1:127.0.0.1:6811/136562268] [v2:127.0.0.1:6812/136562268,v1:127.0.0.1:6813/136562268] exists,up f0ddadd5-e679-4f1d-9dff-ccebbee481e3 2024-04-25T07:14:32.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:14:32.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:14:32.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:14:32.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:14:32.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-04-25T07:14:33.295 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already in. 2024-04-25T07:14:33.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:14:33.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-04-25T07:14:33.860 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already in. 2024-04-25T07:14:33.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:14:33.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-04-25T07:14:33.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:14:33.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:14:33.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:14:33.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:14:33.877 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:14:33.877 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:14:33.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:14:33.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:14:33.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:14:33.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:14:33.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:14:33.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:14:33.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:14:33.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:14:33.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:14:34.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:14:34.390 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:14:34.390 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:14:34.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:14:34.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:34.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:14:34.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822660 2024-04-25T07:14:34.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822660 2024-04-25T07:14:34.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660' 2024-04-25T07:14:34.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:34.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:14:34.702 INFO:tasks.workunit.client.0.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:14:34.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:14:34.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:14:34.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:14:34.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:34.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:14:34.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757251 2024-04-25T07:14:34.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757251 2024-04-25T07:14:34.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-04-25T07:14:34.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:34.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-154618822660 2024-04-25T07:14:34.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:34.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:14:34.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822660 2024-04-25T07:14:34.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:34.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822660 2024-04-25T07:14:34.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-04-25T07:14:34.871 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 154618822660 2024-04-25T07:14:34.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:14:35.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822660 -lt 154618822660 2024-04-25T07:14:35.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:35.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-163208757251 2024-04-25T07:14:35.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:35.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:14:35.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757251 2024-04-25T07:14:35.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:35.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757251 2024-04-25T07:14:35.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-04-25T07:14:35.292 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 163208757251 2024-04-25T07:14:35.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:14:35.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757249 -lt 163208757251 2024-04-25T07:14:35.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:14:36.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:14:36.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:14:37.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757251 -lt 163208757251 2024-04-25T07:14:37.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:14:37.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:37.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:37.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:14:37.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:14:37.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:14:37.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:14:37.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:14:37.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:14:37.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:14:37.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:14:38.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:14:38.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:14:38.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:38.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:38.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:14:38.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:14:38.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:14:38.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:14:38.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:14:38.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:14:38.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:14:38.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:14:38.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:14:38.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:14:38.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:14:38.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:14:38.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:14:38.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:14:39.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:14:39.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:14:39.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:39.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:39.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:14:39.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:14:39.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:14:39.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:14:39.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:14:39.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:14:39.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:14:39.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:14:39.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:14:40.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:14:40.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:14:40.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:14:40.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:14:40.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:14:40.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:14:40.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:14:40.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:14:40.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:14:40.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:14:40.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:14:40.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:14:40.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:14:40.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:14:40.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:14:40.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:40.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:41.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:14:41.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:14:41.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:14:41.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:14:41.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:14:41.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:14:41.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:14:41.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:14:41.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:14:42.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:14:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:14:42.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:14:42.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:14:42.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:14:42.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:14:42.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:14:42.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:14:42.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:14:42.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:14:42.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:14:42.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:14:42.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:42.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:43.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:14:43.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:14:43.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:14:43.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:14:43.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:14:43.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:14:43.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:14:43.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:14:43.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:14:43.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:14:43.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:14:43.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:14:43.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:14:43.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:14:44.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:14:44.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:14:44.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:14:44.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:14:44.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:14:44.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:14:44.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:14:44.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:14:45.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:14:45.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:14:45.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:45.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:45.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:14:45.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:14:45.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:14:45.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:14:45.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:14:45.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:14:45.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:14:45.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:14:45.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:14:46.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:14:46.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:14:46.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:14:46.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:14:46.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:14:47.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:14:47.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:14:47.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:14:47.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:14:47.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:14:47.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:14:47.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:14:47.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:14:48.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:14:48.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:14:48.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:48.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:48.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:14:48.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:14:48.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:14:48.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:14:48.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:14:48.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:14:48.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:14:48.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:14:48.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:14:49.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:14:49.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:14:49.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:14:49.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:14:49.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:14:52.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:14:52.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:14:52.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:14:52.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:14:52.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:14:52.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:14:52.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:14:52.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:14:53.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:14:53.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:14:53.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:53.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:53.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:14:53.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:14:53.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:14:53.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:14:53.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:14:53.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:14:53.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:14:53.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:14:53.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:14:54.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:14:54.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:14:54.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:14:54.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:14:54.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:15:00.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:00.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:00.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:00.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:00.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:00.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:00.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:00.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:00.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:00.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:00.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:00.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:01.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:01.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:01.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:01.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:01.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:01.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:01.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:01.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:01.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:15:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:15:09.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:09.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:09.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:09.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:09.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:09.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:09.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:09.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:09.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:09.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:09.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:09.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:10.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:10.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:10.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:10.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:10.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:10.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:10.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:10.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:10.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:10.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:10.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:10.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:15:10.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:15:11.406 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_fingerprint": "9e0a96f3-c5f4-4063-95a0-83536434a4b0", 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "timestamp": "2024-04-25T07:15:11.408154+0000", 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "tag": "", 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "health": { 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "status": "HEALTH_WARN", 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "checks": { 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:15:11.407 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 17 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [], 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: "OSD_DOWN": { 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 osds down", 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "osd.1 (root=default,host=smithi102) is down" 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.408 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: "PG_DEGRADED": { 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:15:11.409 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.410 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: "mutes": [] 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_first_committed": 1, 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_last_committed": 1, 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap": { 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:11:38.282714Z", 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:11:38.282714Z", 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release": 19, 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release_name": "squid", 2024-04-25T07:15:11.411 INFO:tasks.workunit.client.0.smithi102.stdout: "election_strategy": 1, 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "disallowed_leaders: ": "", 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": false, 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "tiebreaker_mon": "", 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_ranks: ": "", 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "features": { 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "persistent": [ 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "kraken", 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "luminous", 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "mimic", 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap-prune", 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "nautilus", 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "octopus", 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "pacific", 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "elector-pinging", 2024-04-25T07:15:11.412 INFO:tasks.workunit.client.0.smithi102.stdout: "quincy", 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: "reef", 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: "squid" 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: "optional": [] 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: "mons": [ 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: "rank": 0, 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "a", 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.413 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 0 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 0, 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 0, 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_location": "{}" 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: "quorum": [ 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: 0 2024-04-25T07:15:11.414 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap": { 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 41, 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:11:39.322867+0000", 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:14:34.160017+0000", 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "last_up_change": "2024-04-25T07:14:31.602865+0000", 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "last_in_change": "2024-04-25T07:14:33.699518+0000", 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_num": 5800384, 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_set": [ 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "nodown", 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "noin", 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "noout", 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "pglog_hardlimit", 2024-04-25T07:15:11.415 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snapdirs", 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "recovery_deletes", 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "sortbitwise" 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_version": 9, 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_snapshot": "", 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_max": 1, 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "max_osd": 3, 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "min_compat_client": "jewel", 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "require_osd_release": "squid", 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "allow_crimson": false, 2024-04-25T07:15:11.416 INFO:tasks.workunit.client.0.smithi102.stdout: "pools": [ 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_name": "test", 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "create_time": "2024-04-25T07:13:31.435997+0000", 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": 1, 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_names": "hashpspool", 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 3, 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "min_size": 2, 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_rule": 0, 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "object_hash": 2, 2024-04-25T07:15:11.417 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num": 1, 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num": 1, 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num_target": 1, 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_target": 1, 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_pending": 1, 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pg_merge_meta": { 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "source_pgid": "0.0", 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "ready_epoch": 0, 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_started": 0, 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": 0, 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "source_version": "0'0", 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "target_version": "0'0" 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.418 INFO:tasks.workunit.client.0.smithi102.stdout: "last_change": "29", 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend": "0", 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "auid": 0, 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_seq": 0, 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_epoch": 0, 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_snaps": [], 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps": "[]", 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_bytes": 0, 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_objects": 0, 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "tiers": [], 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "tier_of": -1, 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "read_tier": -1, 2024-04-25T07:15:11.419 INFO:tasks.workunit.client.0.smithi102.stdout: "write_tier": -1, 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_mode": "none", 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_bytes": 0, 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_objects": 0, 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_flush_age": 0, 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_evict_age": 0, 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profile": "", 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_params": { 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "none" 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_period": 0, 2024-04-25T07:15:11.420 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_count": 0, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "use_gmt_hitset": true, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "grade_table": [], 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "stripe_width": 0, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_num_objects": 0, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "fast_read": false, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "options": {}, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "application_metadata": {}, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "read_balance": { 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "score_type": "Fair distribution", 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "score_acting": 2.9900000095367432, 2024-04-25T07:15:11.421 INFO:tasks.workunit.client.0.smithi102.stdout: "score_stable": 2.9900000095367432, 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_acting": 2, 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_stable": 2, 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity": 1, 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: "osds": [ 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "b0e4773f-42a0-4d5c-b371-232090487d33", 2024-04-25T07:15:11.422 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 5, 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 29, 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 36, 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 40, 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 30, 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.423 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:15:11.424 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.425 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:11.426 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6803/2016169313", 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6805/2016169313", 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2016169313", 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2016169313", 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:15:11.427 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "0fa44109-b9ae-4d0d-9eb0-a7b7d85f02fe", 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 0, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 0, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 0, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 10, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 23, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 31, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:15:11.428 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:15:11.429 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.430 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.431 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/2763816297", 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/2763816297", 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2763816297", 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2763816297", 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:15:11.432 INFO:tasks.workunit.client.0.smithi102.stdout: "exists" 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "f0ddadd5-e679-4f1d-9dff-ccebbee481e3", 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 15, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 29, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 38, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 0, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 32, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:15:11.433 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:15:11.434 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.435 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:11.436 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/136562268", 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/136562268", 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/136562268", 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/136562268", 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:15:11.437 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_xinfo": [ 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:14:19.139224+0000", 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:12:14.305769+0000", 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.438 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:14:19.703004+0000", 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:12:51.566421+0000", 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:14:20.293973+0000", 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:15:11.439 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:13:27.538288+0000", 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap": [], 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_items": [], 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_primaries": [], 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_temp": [], 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_temp": [], 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "blocklist": {}, 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "range_blocklist": {}, 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profiles": { 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "default": { 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "k": "2", 2024-04-25T07:15:11.440 INFO:tasks.workunit.client.0.smithi102.stdout: "m": "2", 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "plugin": "jerasure", 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "technique": "reed_sol_van" 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps_queue": [], 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "new_removed_snaps": [], 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "new_purged_snaps": [], 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_node_flags": {}, 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "device_class_flags": {}, 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": { 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_enabled": false, 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_bucket_count": 0, 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:15:11.441 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_bucket": 0 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_metadata": [ 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6804/2016169313,v1:127.0.0.1:6805/2016169313]", 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:15:11.442 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:11:46.264592Z", 2024-04-25T07:15:11.443 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6802/2016169313,v1:127.0.0.1:6803/2016169313]", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2016169313,v1:127.0.0.1:6809/2016169313]", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2016169313,v1:127.0.0.1:6807/2016169313]", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:15:11.444 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/2763816297,v1:127.0.0.1:6813/2763816297]", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:15:11.445 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:11.446 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:12:22.508550Z", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/2763816297,v1:127.0.0.1:6811/2763816297]", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2763816297,v1:127.0.0.1:6817/2763816297]", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2763816297,v1:127.0.0.1:6815/2763816297]", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:15:11.447 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:15:11.448 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/136562268,v1:127.0.0.1:6813/136562268]", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:15:11.449 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:12:59.070477Z", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:15:11.450 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/136562268,v1:127.0.0.1:6811/136562268]", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/136562268,v1:127.0.0.1:6817/136562268]", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/136562268,v1:127.0.0.1:6815/136562268]", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:15:11.451 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_clean_epochs": { 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": { 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: "per_pool": [] 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_epochs": [ 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 36 2024-04-25T07:15:11.452 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 38 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_first_committed": 1, 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_last_committed": 41, 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: "crushmap": { 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": [ 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.0", 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.453 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.1", 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.2", 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: "types": [ 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.454 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 0, 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd" 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "host" 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 2, 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "chassis" 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 3, 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "rack" 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.455 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 4, 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "row" 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 5, 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pdu" 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 6, 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pod" 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 7, 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "room" 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 8, 2024-04-25T07:15:11.456 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "datacenter" 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 9, 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "zone" 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 10, 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "region" 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "root" 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: "buckets": [ 2024-04-25T07:15:11.457 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -1, 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default", 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.458 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -2, 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default~hdd", 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.459 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102", 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.460 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102~hdd", 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:15:11.461 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.462 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: "rules": [ 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_id": 0, 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_name": "replicated_rule", 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: "steps": [ 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "take", 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: "item": -1, 2024-04-25T07:15:11.463 INFO:tasks.workunit.client.0.smithi102.stdout: "item_name": "default" 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "choose_firstn", 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 0, 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "osd" 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "emit" 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: "tunables": { 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_tries": 0, 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_total_tries": 50, 2024-04-25T07:15:11.464 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_stable": 1, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_descents": 100, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_collision_tries": 100, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "straw_calc_version": 1, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "profile": "jewel", 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_tunables": 1, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "legacy_tunables": 0, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "minimum_required_version": "jewel", 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables": 1, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables2": 1, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v2_rules": 0, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables3": 1, 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v3_rules": 0, 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v4_buckets": 1, 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables5": 1, 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v5_rules": 0, 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "has_msr_rules": 0 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_args": {} 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "fsmap": { 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "default_fscid": -1, 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": { 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": {}, 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "ro_compat": {}, 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "incompat": { 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_1": "base v0.20", 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_8": "no anchor table", 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_9": "file layout v2", 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_11": "minor log segments", 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_flags": { 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi102.stdout: "enable_multiple": true, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "ever_enabled_multiple": true 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "standbys": [], 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "filesystems": [] 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "auth": { 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 5, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "num_secrets": 3 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 1, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_active": 1, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_unknown": 0, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osd": 3, 2024-04-25T07:15:11.468 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_sum": { 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 61700, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 100, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 300, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 100, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 100, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:15:11.469 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 100, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 100, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:15:11.470 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:15:11.471 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 0, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 0, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 16, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 16, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 2, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 2, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 0 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.472 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum": { 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 3, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83620, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2104, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314489180, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:15:11.473 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036920320, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2154496, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 459927, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10396, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416932 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:15:11.474 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum_by_class": { 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "hdd": { 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:15:11.475 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 3, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83620, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2104, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314489180, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036920320, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2154496, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 459927, 2024-04-25T07:15:11.476 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10396, 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416932 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:15:11.477 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_stats": [ 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 1, 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:15:11.478 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 61700, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 100, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 300, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 100, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 100, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 100, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 100, 2024-04-25T07:15:11.479 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:15:11.480 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 1220608, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 184566, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:15:11.481 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 16, 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 16, 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 2, 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 2, 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 3 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_stats": [ 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:15:11.482 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 154618822667 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 42949672989 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 163208757258 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_state": [ 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "active+undersized+degraded", 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 1 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.483 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_osd": [ 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 1, 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:15:11.484 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 0, 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": [ 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": {} 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: "servicemap": { 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 5, 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:14:22.755420+0000", 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: "services": {} 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_last_committed": 95, 2024-04-25T07:15:11.485 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_first_committed": 1, 2024-04-25T07:15:11.486 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_last_committed": 376, 2024-04-25T07:15:11.486 INFO:tasks.workunit.client.0.smithi102.stdout: "paxos": { 2024-04-25T07:15:11.486 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:15:11.486 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 520, 2024-04-25T07:15:11.486 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pn": 0, 2024-04-25T07:15:11.486 INFO:tasks.workunit.client.0.smithi102.stdout: "accepted_pn": 0 2024-04-25T07:15:11.486 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:11.486 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:15:11.487 INFO:tasks.workunit.client.0.smithi102.stderr:report 1769144714 2024-04-25T07:15:11.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:15:11.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-04-25T07:15:11.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-04-25T07:15:11.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-04-25T07:15:11.487 INFO:tasks.workunit.client.0.smithi102.stdout:writing non-divergent object existing_4 2024-04-25T07:15:11.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:15:11.833 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:15:11.833 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:14:34.504019+0000 29'100 41:128 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:28:38.836958+0000 0 0 2024-04-25T07:15:11.833 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:15:11.833 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:15:11.834 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:15:11.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:15:11.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:15:11.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-04-25T07:15:11.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:15:11.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:15:11.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:15:11.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:15:11.976 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:15:11.976 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:15:11.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:15:11.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:15:11.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:15:12.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:15:12.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:15:12.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:15:12.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:15:12.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:15:12.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:15:12.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:15:12.497 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:15:12.497 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:15:12.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:15:12.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:15:12.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:15:12.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822670 2024-04-25T07:15:12.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822670 2024-04-25T07:15:12.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822670' 2024-04-25T07:15:12.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:15:12.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:15:12.889 INFO:tasks.workunit.client.0.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:15:12.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:15:12.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:15:12.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:15:12.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:15:12.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:15:13.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757261 2024-04-25T07:15:13.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757261 2024-04-25T07:15:13.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822670 2-163208757261' 2024-04-25T07:15:13.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:15:13.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-154618822670 2024-04-25T07:15:13.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:15:13.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:15:13.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822670 2024-04-25T07:15:13.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:15:13.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822670 2024-04-25T07:15:13.054 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 154618822670 2024-04-25T07:15:13.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 154618822670' 2024-04-25T07:15:13.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:15:13.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822670 -lt 154618822670 2024-04-25T07:15:13.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:15:13.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-163208757261 2024-04-25T07:15:13.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:15:13.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:15:13.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757261 2024-04-25T07:15:13.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:15:13.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757261 2024-04-25T07:15:13.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 163208757261' 2024-04-25T07:15:13.479 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 163208757261 2024-04-25T07:15:13.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:15:13.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757259 -lt 163208757261 2024-04-25T07:15:13.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:15:14.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:15:14.896 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:15:15.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757261 -lt 163208757261 2024-04-25T07:15:15.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:15:15.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:15.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:15.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:15:15.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:15.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:15.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:15.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:15.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:15.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:15.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:16.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:16.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:16.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:16.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:16.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:16.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:15:16.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:15:16.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:15:16.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:16.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:15:16.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:16.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:16.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:16.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:16.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:16.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:16.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:16.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:17.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:17.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:17.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:17.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:17.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:17.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:17.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:17.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:17.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:17.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:17.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:17.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:17.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:18.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:18.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:18.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:15:18.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:18.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:15:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:18.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:18.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:18.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:18.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:18.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:18.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:19.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:19.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:19.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:19.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:19.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:19.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:19.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:19.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:19.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:19.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:19.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:19.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:19.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:20.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:20.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:20.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:15:20.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:20.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:15:20.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:20.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:20.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:20.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:20.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:20.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:20.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:20.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:20.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:20.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:20.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:20.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:21.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:21.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:21.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:21.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:21.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:21.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:21.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:21.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:21.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:22.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:22.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:22.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:15:22.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:22.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:15:22.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:22.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:22.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:22.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:22.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:22.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:22.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:22.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:23.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:23.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:23.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:23.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:23.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:23.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:23.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:23.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:23.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:23.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:23.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:23.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:24.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:24.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:24.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:15:24.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:24.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:15:25.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:25.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:25.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:25.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:25.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:25.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:25.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:25.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:26.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:26.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:26.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:26.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:26.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:26.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:26.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:26.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:26.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:26.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:26.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:26.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:26.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:27.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:27.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:27.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:15:27.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:27.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:15:30.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:30.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:30.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:30.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:30.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:30.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:30.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:30.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:31.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:31.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:31.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:31.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:31.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:31.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:31.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:31.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:31.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:31.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:31.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:31.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:31.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:32.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:32.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:32.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:15:32.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:32.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:15:38.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:38.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:38.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:38.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:38.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:38.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:38.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:38.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:39.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:39.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:39.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:39.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:39.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:39.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:39.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:39.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:39.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:39.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:39.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:39.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:40.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:40.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:40.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:15:40.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:40.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:15:47.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:47.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:47.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:47.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:47.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:47.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:47.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:47.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:47.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:47.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:47.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:47.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:48.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:48.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:48.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:48.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:48.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:48.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:48.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:48.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:48.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:48.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:48.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:48.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:15:48.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:15:49.492 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:15:49.492 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_fingerprint": "9e0a96f3-c5f4-4063-95a0-83536434a4b0", 2024-04-25T07:15:49.492 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "timestamp": "2024-04-25T07:15:49.495468+0000", 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "tag": "", 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "health": { 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "status": "HEALTH_WARN", 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "checks": { 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 17 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [], 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:15:49.493 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: "OSD_DOWN": { 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 osds down", 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "osd.1 (root=default,host=smithi102) is down" 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: "PG_DEGRADED": { 2024-04-25T07:15:49.494 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 2 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.0 is stuck undersized for 74s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:15:49.495 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: "mutes": [] 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_first_committed": 1, 2024-04-25T07:15:49.496 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_last_committed": 1, 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap": { 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:11:38.282714Z", 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:11:38.282714Z", 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release": 19, 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release_name": "squid", 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "election_strategy": 1, 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "disallowed_leaders: ": "", 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": false, 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "tiebreaker_mon": "", 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_ranks: ": "", 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "features": { 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "persistent": [ 2024-04-25T07:15:49.497 INFO:tasks.workunit.client.0.smithi102.stdout: "kraken", 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "luminous", 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "mimic", 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap-prune", 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "nautilus", 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "octopus", 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "pacific", 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "elector-pinging", 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "quincy", 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "reef", 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "squid" 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "optional": [] 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.498 INFO:tasks.workunit.client.0.smithi102.stdout: "mons": [ 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: "rank": 0, 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "a", 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 0 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 0, 2024-04-25T07:15:49.499 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 0, 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_location": "{}" 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: "quorum": [ 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: 0 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap": { 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 41, 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:11:39.322867+0000", 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:14:34.160017+0000", 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: "last_up_change": "2024-04-25T07:14:31.602865+0000", 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: "last_in_change": "2024-04-25T07:14:33.699518+0000", 2024-04-25T07:15:49.500 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_num": 5800384, 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_set": [ 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "nodown", 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "noin", 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "noout", 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "pglog_hardlimit", 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snapdirs", 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "recovery_deletes", 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "sortbitwise" 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_version": 9, 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_snapshot": "", 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_max": 1, 2024-04-25T07:15:49.501 INFO:tasks.workunit.client.0.smithi102.stdout: "max_osd": 3, 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "min_compat_client": "jewel", 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "require_osd_release": "squid", 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "allow_crimson": false, 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "pools": [ 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_name": "test", 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "create_time": "2024-04-25T07:13:31.435997+0000", 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": 1, 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_names": "hashpspool", 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 3, 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "min_size": 2, 2024-04-25T07:15:49.502 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_rule": 0, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "object_hash": 2, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num": 1, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num": 1, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num_target": 1, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_target": 1, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_pending": 1, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pg_merge_meta": { 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "source_pgid": "0.0", 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "ready_epoch": 0, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_started": 0, 2024-04-25T07:15:49.503 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": 0, 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "source_version": "0'0", 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "target_version": "0'0" 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "last_change": "29", 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend": "0", 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "auid": 0, 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_seq": 0, 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_epoch": 0, 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_snaps": [], 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps": "[]", 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_bytes": 0, 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_objects": 0, 2024-04-25T07:15:49.504 INFO:tasks.workunit.client.0.smithi102.stdout: "tiers": [], 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "tier_of": -1, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "read_tier": -1, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "write_tier": -1, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_mode": "none", 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_bytes": 0, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_objects": 0, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_flush_age": 0, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_evict_age": 0, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profile": "", 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_params": { 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "none" 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_period": 0, 2024-04-25T07:15:49.505 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_count": 0, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "use_gmt_hitset": true, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "grade_table": [], 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "stripe_width": 0, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_num_objects": 0, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "fast_read": false, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "options": {}, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "application_metadata": {}, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "read_balance": { 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "score_type": "Fair distribution", 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "score_acting": 2.9900000095367432, 2024-04-25T07:15:49.506 INFO:tasks.workunit.client.0.smithi102.stdout: "score_stable": 2.9900000095367432, 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_acting": 2, 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_stable": 2, 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity": 1, 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: "osds": [ 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "b0e4773f-42a0-4d5c-b371-232090487d33", 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:15:49.507 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 5, 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 29, 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 36, 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 40, 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 30, 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.508 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.509 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.510 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2016169313 2024-04-25T07:15:49.511 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6803/2016169313", 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6805/2016169313", 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2016169313", 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2016169313", 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "0fa44109-b9ae-4d0d-9eb0-a7b7d85f02fe", 2024-04-25T07:15:49.512 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 0, 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 0, 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 0, 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 10, 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 23, 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 31, 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.513 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.514 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:15:49.515 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2763816297 2024-04-25T07:15:49.516 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/2763816297", 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/2763816297", 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2763816297", 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2763816297", 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: "exists" 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "f0ddadd5-e679-4f1d-9dff-ccebbee481e3", 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:15:49.517 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 15, 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 29, 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 38, 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 0, 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 32, 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:49.518 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.519 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:49.520 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.521 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 136562268 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/136562268", 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/136562268", 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/136562268", 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/136562268", 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:15:49.522 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_xinfo": [ 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:14:19.139224+0000", 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:12:14.305769+0000", 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.523 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:14:19.703004+0000", 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:12:51.566421+0000", 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:14:20.293973+0000", 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:15:49.524 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:13:27.538288+0000", 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap": [], 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_items": [], 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_primaries": [], 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_temp": [], 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_temp": [], 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "blocklist": {}, 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "range_blocklist": {}, 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profiles": { 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "default": { 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "k": "2", 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "m": "2", 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "plugin": "jerasure", 2024-04-25T07:15:49.525 INFO:tasks.workunit.client.0.smithi102.stdout: "technique": "reed_sol_van" 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps_queue": [], 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "new_removed_snaps": [], 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "new_purged_snaps": [], 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_node_flags": {}, 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "device_class_flags": {}, 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": { 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_enabled": false, 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_bucket_count": 0, 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_bucket": 0 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.526 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_metadata": [ 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6804/2016169313,v1:127.0.0.1:6805/2016169313]", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:15:49.527 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:11:46.264592Z", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:15:49.528 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6802/2016169313,v1:127.0.0.1:6803/2016169313]", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2016169313,v1:127.0.0.1:6809/2016169313]", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2016169313,v1:127.0.0.1:6807/2016169313]", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:15:49.529 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/2763816297,v1:127.0.0.1:6813/2763816297]", 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:15:49.530 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:15:49.531 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:12:22.508550Z", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/2763816297,v1:127.0.0.1:6811/2763816297]", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2763816297,v1:127.0.0.1:6817/2763816297]", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2763816297,v1:127.0.0.1:6815/2763816297]", 2024-04-25T07:15:49.532 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.533 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/136562268,v1:127.0.0.1:6813/136562268]", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:15:49.534 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:12:59.070477Z", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:15:49.535 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/136562268,v1:127.0.0.1:6811/136562268]", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/136562268,v1:127.0.0.1:6817/136562268]", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/136562268,v1:127.0.0.1:6815/136562268]", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:15:49.536 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_clean_epochs": { 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": { 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: "per_pool": [] 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_epochs": [ 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 36 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.537 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 38 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_first_committed": 1, 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_last_committed": 41, 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: "crushmap": { 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": [ 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.0", 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.538 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.1", 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.2", 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: "types": [ 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 0, 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd" 2024-04-25T07:15:49.539 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "host" 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 2, 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "chassis" 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 3, 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "rack" 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 4, 2024-04-25T07:15:49.540 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "row" 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 5, 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pdu" 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 6, 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pod" 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 7, 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "room" 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.541 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 8, 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "datacenter" 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 9, 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "zone" 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 10, 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "region" 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "root" 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.542 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "buckets": [ 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -1, 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default", 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:15:49.543 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -2, 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default~hdd", 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:49.544 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102", 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:15:49.545 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:15:49.546 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102~hdd", 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.547 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: "rules": [ 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_id": 0, 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_name": "replicated_rule", 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: "steps": [ 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "take", 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: "item": -1, 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: "item_name": "default" 2024-04-25T07:15:49.548 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "choose_firstn", 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 0, 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "osd" 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "emit" 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: "tunables": { 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_tries": 0, 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_total_tries": 50, 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:15:49.549 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_stable": 1, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_descents": 100, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_collision_tries": 100, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "straw_calc_version": 1, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "profile": "jewel", 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_tunables": 1, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "legacy_tunables": 0, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "minimum_required_version": "jewel", 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables": 1, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables2": 1, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v2_rules": 0, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables3": 1, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v3_rules": 0, 2024-04-25T07:15:49.550 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v4_buckets": 1, 2024-04-25T07:15:49.551 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables5": 1, 2024-04-25T07:15:49.551 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v5_rules": 0, 2024-04-25T07:15:49.551 INFO:tasks.workunit.client.0.smithi102.stdout: "has_msr_rules": 0 2024-04-25T07:15:49.551 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.551 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_args": {} 2024-04-25T07:15:49.551 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.551 INFO:tasks.workunit.client.0.smithi102.stdout: "fsmap": { 2024-04-25T07:15:49.551 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:15:49.551 INFO:tasks.workunit.client.0.smithi102.stdout: "default_fscid": -1, 2024-04-25T07:15:49.551 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": { 2024-04-25T07:15:49.551 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": {}, 2024-04-25T07:15:49.552 INFO:tasks.workunit.client.0.smithi102.stderr:report 1501906036 2024-04-25T07:15:49.552 INFO:tasks.workunit.client.0.smithi102.stdout: "ro_compat": {}, 2024-04-25T07:15:49.552 INFO:tasks.workunit.client.0.smithi102.stdout: "incompat": { 2024-04-25T07:15:49.552 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_1": "base v0.20", 2024-04-25T07:15:49.552 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:15:49.552 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:15:49.552 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:15:49.552 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:15:49.552 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:15:49.552 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_8": "no anchor table", 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_9": "file layout v2", 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_11": "minor log segments", 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_flags": { 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: "enable_multiple": true, 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: "ever_enabled_multiple": true 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: "standbys": [], 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: "filesystems": [] 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:15:49.553 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "auth": { 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 5, 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "num_secrets": 3 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 1, 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_active": 1, 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_unknown": 0, 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osd": 3, 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_sum": { 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 62050, 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 100, 2024-04-25T07:15:49.554 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 300, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 100, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 100, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 101, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 101, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:15:49.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:15:49.556 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 0, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 0, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:49.557 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 17, 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 17, 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 2, 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 2, 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 0 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum": { 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 3, 2024-04-25T07:15:49.558 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83620, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2104, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314489180, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036920320, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2154496, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 460627, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:15:49.559 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10396, 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416932 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:15:49.560 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum_by_class": { 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "hdd": { 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 3, 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:15:49.561 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83620, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2104, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314489180, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036920320, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2154496, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 460627, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:15:49.562 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10396, 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416932 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:15:49.563 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_stats": [ 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 1, 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 62050, 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 100, 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 300, 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 100, 2024-04-25T07:15:49.564 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 100, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 101, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 101, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:15:49.565 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:15:49.566 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 1220608, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 185266, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 17, 2024-04-25T07:15:49.567 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 17, 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 2, 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 2, 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 3 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_stats": [ 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 154618822677 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 42949672989 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:15:49.568 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 163208757268 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_state": [ 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "active+undersized+degraded", 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 1 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_osd": [ 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.569 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 1, 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 0, 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": [ 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:15:49.570 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": {} 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "servicemap": { 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 7, 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:15:32.772922+0000", 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "services": {} 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_last_committed": 114, 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_first_committed": 1, 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_last_committed": 443, 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "paxos": { 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 606, 2024-04-25T07:15:49.571 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pn": 0, 2024-04-25T07:15:49.572 INFO:tasks.workunit.client.0.smithi102.stdout: "accepted_pn": 0 2024-04-25T07:15:49.572 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:15:49.572 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:15:49.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:15:49.572 INFO:tasks.workunit.client.0.smithi102.stdout:delay recovery 2024-04-25T07:15:49.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-04-25T07:15:49.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:15:49.889 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:15:49.890 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:14:34.504019+0000 41'101 41:129 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:28:38.836958+0000 0 0 2024-04-25T07:15:49.890 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:15:49.890 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:15:49.890 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:15:49.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:15:49.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-04-25T07:15:49.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:15:49.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:15:49.906 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:15:49.906 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:15:49.907 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:15:49.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.0.asok 2024-04-25T07:15:49.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:15:49.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:15:50.014 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:15:50.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:15:50.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-04-25T07:15:50.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:15:50.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:15:50.028 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:15:50.028 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:15:50.028 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:15:50.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.2.asok 2024-04-25T07:15:50.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:15:50.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:15:50.138 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:15:50.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-04-25T07:15:50.151 INFO:tasks.workunit.client.0.smithi102.stdout:revive divergent 1 2024-04-25T07:15:50.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:15:50.558 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:15:50.558 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:14:34.504019+0000 41'101 41:129 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:28:38.836958+0000 0 0 2024-04-25T07:15:50.558 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:15:50.558 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:15:50.558 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:15:50.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-04-25T07:15:51.214 INFO:tasks.workunit.client.0.smithi102.stderr:noup is set 2024-04-25T07:15:51.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-04-25T07:15:51.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:15:51.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:15:51.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:15:51.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:15:51.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:15:51.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:15:51.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:15:51.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:15:51.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:15:51.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:15:51.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:15:51.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:15:51.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:15:51.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:15:51.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:15:51.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:15:51.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:15:51.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:15:51.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:15:51.237 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:15:51.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:15:51.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:15:51.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:15:51.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:15:51.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:15:51.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:15:51.276 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:15:51.279+0000 7f97f80bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:51.281 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:15:51.283+0000 7f97f80bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:51.282 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:15:51.283+0000 7f97f80bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:51.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:15:51.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-04-25T07:15:51.817 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:15:51.819+0000 7f97f80bc7c0 -1 Falling back to public interface 2024-04-25T07:15:54.749 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:15:54.751+0000 7f97f80bc7c0 -1 osd.1 29 log_to_monitors true 2024-04-25T07:15:56.655 INFO:tasks.workunit.client.0.smithi102.stdout:delay recovery divergent 2024-04-25T07:15:56.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-04-25T07:15:56.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:15:57.062 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:15:57.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:14:34.504019+0000 41'101 43:133 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:28:38.836958+0000 0 0 2024-04-25T07:15:57.062 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:15:57.062 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:15:57.063 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:15:57.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-04-25T07:15:57.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:15:57.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:15:57.080 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:15:57.080 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:15:57.080 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:15:57.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.1.asok 2024-04-25T07:15:57.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:15:57.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:15:57.179 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:15:57.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-04-25T07:15:57.877 INFO:tasks.workunit.client.0.smithi102.stderr:noup is unset 2024-04-25T07:15:57.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-04-25T07:15:57.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:15:57.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:15:57.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:15:57.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:15:57.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:57.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:15:57.898 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:15:57.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:57.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:15:58.497 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/2016169313,v1:127.0.0.1:6803/2016169313] [v2:127.0.0.1:6804/2016169313,v1:127.0.0.1:6805/2016169313] exists,up b0e4773f-42a0-4d5c-b371-232090487d33 2024-04-25T07:15:58.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:15:58.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:15:58.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:15:58.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-04-25T07:15:58.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:15:58.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:15:58.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:15:58.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:15:58.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:58.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:15:58.499 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:15:58.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:58.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:15:58.952 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/879082079,v1:127.0.0.1:6819/879082079] [v2:127.0.0.1:6820/879082079,v1:127.0.0.1:6821/879082079] exists,up 0fa44109-b9ae-4d0d-9eb0-a7b7d85f02fe 2024-04-25T07:15:58.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:15:58.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:15:58.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:15:58.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-04-25T07:15:58.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:15:58.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:15:58.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:15:58.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:15:58.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:58.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:15:58.954 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:15:58.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:58.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:59.457 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/136562268,v1:127.0.0.1:6811/136562268] [v2:127.0.0.1:6812/136562268,v1:127.0.0.1:6813/136562268] exists,up f0ddadd5-e679-4f1d-9dff-ccebbee481e3 2024-04-25T07:15:59.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:15:59.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:15:59.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:15:59.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:15:59.839 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:15:59.839 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:14:34.504019+0000 41'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:28:38.836958+0000 0 0 2024-04-25T07:15:59.839 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:15:59.839 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:15:59.839 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:15:59.853 INFO:tasks.workunit.client.0.smithi102.stdout:wait for peering 2024-04-25T07:15:59.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-04-25T07:15:59.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:00.234 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:00.234 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:14:34.504019+0000 41'101 44:135 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:28:38.836958+0000 0 0 2024-04-25T07:16:00.234 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:16:00.234 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:00.235 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:16:00.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-04-25T07:16:00.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-04-25T07:16:00.398 INFO:tasks.workunit.client.0.smithi102.stdout:killing divergent 1 2024-04-25T07:16:00.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:00.796 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:00.796 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 4 4 0 0 62050 0 0 11 90 11 active+recovering+degraded 2024-04-25T07:15:59.093298+0000 41'101 47:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:16:00.796 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:16:00.797 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:00.797 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:16:00.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:16:00.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:16:00.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:16:00.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:16:00.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:16:00.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:16:00.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:16:00.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.26156.out 2024-04-25T07:16:00.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.26156.out 2024-04-25T07:16:00.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:16:00.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:16:00.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:16:00.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:16:00.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:16:00.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.26156.out 2024-04-25T07:16:02.300 INFO:tasks.workunit.client.0.smithi102.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-04-25T07:16:02.302 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:023dd3d2:::existing_12:head# 2024-04-25T07:16:02.302 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:04c40a0c:::existing_93:head# 2024-04-25T07:16:02.303 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:064ff5fb:::existing_36:head# 2024-04-25T07:16:02.304 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:091f9f1d:::existing_94:head# 2024-04-25T07:16:02.304 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0a865a93:::existing_54:head# 2024-04-25T07:16:02.305 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0fed513c:::existing_41:head# 2024-04-25T07:16:02.305 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-04-25T07:16:02.306 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:169157fa:::existing_81:head# 2024-04-25T07:16:02.306 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:17731382:::existing_58:head# 2024-04-25T07:16:02.307 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:18a45a39:::existing_43:head# 2024-04-25T07:16:02.308 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1c160f01:::existing_66:head# 2024-04-25T07:16:02.308 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1dd6c249:::existing_97:head# 2024-04-25T07:16:02.309 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:227e0c77:::existing_7:head# 2024-04-25T07:16:02.309 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:23e1b063:::existing_59:head# 2024-04-25T07:16:02.310 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:28566edb:::existing_78:head# 2024-04-25T07:16:02.310 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2a101182:::existing_20:head# 2024-04-25T07:16:02.311 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2d1519da:::existing_96:head# 2024-04-25T07:16:02.311 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3078b87a:::existing_86:head# 2024-04-25T07:16:02.312 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:30bde01b:::existing_15:head# 2024-04-25T07:16:02.313 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:317a7e19:::existing_56:head# 2024-04-25T07:16:02.313 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-04-25T07:16:02.314 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3e16d84b:::existing_48:head# 2024-04-25T07:16:02.314 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:45fe8203:::existing_75:head# 2024-04-25T07:16:02.315 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-04-25T07:16:02.315 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5298dcdb:::existing_57:head# 2024-04-25T07:16:02.316 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:52ebaa44:::existing_32:head# 2024-04-25T07:16:02.316 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:53588912:::existing_9:head# 2024-04-25T07:16:02.317 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-04-25T07:16:02.317 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5da0255c:::existing_13:head# 2024-04-25T07:16:02.318 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5da2dab8:::existing_53:head# 2024-04-25T07:16:02.319 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5ee3514b:::existing_30:head# 2024-04-25T07:16:02.319 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:602f83fe:::foo:head# 2024-04-25T07:16:02.320 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:60426cc7:::existing_62:head# 2024-04-25T07:16:02.320 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:628918ca:::existing_79:head# 2024-04-25T07:16:02.320 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-04-25T07:16:02.321 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:67e1780f:::existing_27:head# 2024-04-25T07:16:02.321 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:68143ca1:::existing_71:head# 2024-04-25T07:16:02.322 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6bac9509:::existing_29:head# 2024-04-25T07:16:02.323 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6bbec798:::existing_45:head# 2024-04-25T07:16:02.323 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6cb23a40:::existing_65:head# 2024-04-25T07:16:02.324 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7055ec5e:::existing_83:head# 2024-04-25T07:16:02.324 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:70a54e2b:::existing_87:head# 2024-04-25T07:16:02.324 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:750ace21:::existing_19:head# 2024-04-25T07:16:02.325 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:78623b1d:::existing_6:head# 2024-04-25T07:16:02.325 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:79881152:::existing_63:head# 2024-04-25T07:16:02.326 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-04-25T07:16:02.326 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:81490677:::existing_76:head# 2024-04-25T07:16:02.327 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-04-25T07:16:02.327 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-04-25T07:16:02.328 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-04-25T07:16:02.328 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:923c5d4d:::existing_5:head# 2024-04-25T07:16:02.329 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:92731e71:::existing_89:head# 2024-04-25T07:16:02.329 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:928205cf:::existing_82:head# 2024-04-25T07:16:02.330 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:93be2267:::existing_24:head# 2024-04-25T07:16:02.330 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:99ceda2c:::existing_44:head# 2024-04-25T07:16:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9dd917ee:::existing_26:head# 2024-04-25T07:16:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a28805a6:::existing_72:head# 2024-04-25T07:16:02.332 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a294a8f6:::existing_16:head# 2024-04-25T07:16:02.332 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a2f5473a:::existing_42:head# 2024-04-25T07:16:02.333 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a504a533:::existing_90:head# 2024-04-25T07:16:02.333 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a691337e:::existing_61:head# 2024-04-25T07:16:02.333 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-04-25T07:16:02.334 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a9fde86e:::existing_18:head# 2024-04-25T07:16:02.334 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:aa309faa:::existing_68:head# 2024-04-25T07:16:02.335 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-04-25T07:16:02.335 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b279b1fe:::existing_10:head# 2024-04-25T07:16:02.336 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b343e277:::existing_88:head# 2024-04-25T07:16:02.336 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b7130e90:::existing_80:head# 2024-04-25T07:16:02.337 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bc6f7309:::existing_70:head# 2024-04-25T07:16:02.338 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:be8489b7:::existing_38:head# 2024-04-25T07:16:02.338 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-04-25T07:16:02.339 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bebec877:::existing_74:head# 2024-04-25T07:16:02.339 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c04347d8:::existing_60:head# 2024-04-25T07:16:02.340 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c156033e:::existing_100:head# 2024-04-25T07:16:02.340 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-04-25T07:16:02.341 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-04-25T07:16:02.341 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-04-25T07:16:02.342 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-04-25T07:16:02.342 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ccb3865b:::existing_73:head# 2024-04-25T07:16:02.343 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cd84e19e:::existing_40:head# 2024-04-25T07:16:02.343 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-04-25T07:16:02.344 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d41a25c9:::existing_22:head# 2024-04-25T07:16:02.345 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d7f088da:::existing_91:head# 2024-04-25T07:16:02.345 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:16:02.346 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-04-25T07:16:02.346 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dd677bd3:::existing_35:head# 2024-04-25T07:16:02.347 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-04-25T07:16:02.347 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e0e22077:::existing_21:head# 2024-04-25T07:16:02.348 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e255b710:::existing_55:head# 2024-04-25T07:16:02.348 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:eff1d5db:::existing_14:head# 2024-04-25T07:16:02.349 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f1b156bf:::existing_23:head# 2024-04-25T07:16:02.349 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f5ce4199:::existing_34:head# 2024-04-25T07:16:02.350 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f6809456:::existing_11:head# 2024-04-25T07:16:02.350 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f699a2a9:::existing_67:head# 2024-04-25T07:16:02.351 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f9baac43:::existing_25:head# 2024-04-25T07:16:02.351 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fb76688e:::existing_51:head# 2024-04-25T07:16:02.352 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fbb92eec:::existing_69:head# 2024-04-25T07:16:02.352 INFO:tasks.workunit.client.0.smithi102.stderr:Export successful 2024-04-25T07:16:02.352 INFO:tasks.workunit.client.0.smithi102.stdout: marking collection for removal 2024-04-25T07:16:02.352 INFO:tasks.workunit.client.0.smithi102.stdout:setting '_remove' omap key 2024-04-25T07:16:02.352 INFO:tasks.workunit.client.0.smithi102.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T07:16:02.566 INFO:tasks.workunit.client.0.smithi102.stderr:Remove successful 2024-04-25T07:16:02.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.26156.out 2024-04-25T07:16:02.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:16:02.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:16:02.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:16:02.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:16:02.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:16:02.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.26156.out 2024-04-25T07:16:03.840 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={23=1}} deleted_pools ) 2024-04-25T07:16:03.840 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.0 2024-04-25T07:16:03.840 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:023dd3d2:::existing_12:head# 2024-04-25T07:16:03.840 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:03.920 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:04c40a0c:::existing_93:head# 2024-04-25T07:16:03.920 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:03.962 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:064ff5fb:::existing_36:head# 2024-04-25T07:16:03.962 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.004 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:091f9f1d:::existing_94:head# 2024-04-25T07:16:04.004 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.045 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0a865a93:::existing_54:head# 2024-04-25T07:16:04.045 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.087 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0fed513c:::existing_41:head# 2024-04-25T07:16:04.087 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.129 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-04-25T07:16:04.129 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.171 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:169157fa:::existing_81:head# 2024-04-25T07:16:04.171 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.212 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:17731382:::existing_58:head# 2024-04-25T07:16:04.213 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.254 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:18a45a39:::existing_43:head# 2024-04-25T07:16:04.254 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.296 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1c160f01:::existing_66:head# 2024-04-25T07:16:04.296 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.338 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1dd6c249:::existing_97:head# 2024-04-25T07:16:04.338 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.380 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:227e0c77:::existing_7:head# 2024-04-25T07:16:04.380 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.421 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:23e1b063:::existing_59:head# 2024-04-25T07:16:04.422 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.463 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:28566edb:::existing_78:head# 2024-04-25T07:16:04.463 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.505 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2a101182:::existing_20:head# 2024-04-25T07:16:04.505 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.547 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2d1519da:::existing_96:head# 2024-04-25T07:16:04.547 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.589 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3078b87a:::existing_86:head# 2024-04-25T07:16:04.589 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.630 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:30bde01b:::existing_15:head# 2024-04-25T07:16:04.630 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.672 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:317a7e19:::existing_56:head# 2024-04-25T07:16:04.672 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.714 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-04-25T07:16:04.714 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.756 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3e16d84b:::existing_48:head# 2024-04-25T07:16:04.756 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.798 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:45fe8203:::existing_75:head# 2024-04-25T07:16:04.798 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.852 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-04-25T07:16:04.853 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.931 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5298dcdb:::existing_57:head# 2024-04-25T07:16:04.931 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:04.990 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:52ebaa44:::existing_32:head# 2024-04-25T07:16:04.990 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.032 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:53588912:::existing_9:head# 2024-04-25T07:16:05.032 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.074 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-04-25T07:16:05.074 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5da0255c:::existing_13:head# 2024-04-25T07:16:05.115 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.157 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5da2dab8:::existing_53:head# 2024-04-25T07:16:05.157 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.199 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5ee3514b:::existing_30:head# 2024-04-25T07:16:05.199 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.241 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:602f83fe:::foo:head# 2024-04-25T07:16:05.241 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.286 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:60426cc7:::existing_62:head# 2024-04-25T07:16:05.358 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.359 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:628918ca:::existing_79:head# 2024-04-25T07:16:05.359 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.425 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-04-25T07:16:05.425 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.484 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:67e1780f:::existing_27:head# 2024-04-25T07:16:05.484 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.517 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68143ca1:::existing_71:head# 2024-04-25T07:16:05.517 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.550 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6bac9509:::existing_29:head# 2024-04-25T07:16:05.551 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.584 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6bbec798:::existing_45:head# 2024-04-25T07:16:05.584 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.617 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6cb23a40:::existing_65:head# 2024-04-25T07:16:05.617 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.651 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7055ec5e:::existing_83:head# 2024-04-25T07:16:05.651 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.684 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:70a54e2b:::existing_87:head# 2024-04-25T07:16:05.684 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.726 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:750ace21:::existing_19:head# 2024-04-25T07:16:05.726 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.768 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:78623b1d:::existing_6:head# 2024-04-25T07:16:05.768 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.810 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:79881152:::existing_63:head# 2024-04-25T07:16:05.810 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.851 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-04-25T07:16:05.852 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.893 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:81490677:::existing_76:head# 2024-04-25T07:16:05.893 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.935 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-04-25T07:16:05.935 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:05.977 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-04-25T07:16:05.977 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.019 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-04-25T07:16:06.019 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.061 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:923c5d4d:::existing_5:head# 2024-04-25T07:16:06.061 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.102 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:92731e71:::existing_89:head# 2024-04-25T07:16:06.102 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.144 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:928205cf:::existing_82:head# 2024-04-25T07:16:06.144 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.186 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:93be2267:::existing_24:head# 2024-04-25T07:16:06.186 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.228 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:99ceda2c:::existing_44:head# 2024-04-25T07:16:06.228 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.270 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9dd917ee:::existing_26:head# 2024-04-25T07:16:06.270 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.311 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a28805a6:::existing_72:head# 2024-04-25T07:16:06.312 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.370 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a294a8f6:::existing_16:head# 2024-04-25T07:16:06.370 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.412 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a2f5473a:::existing_42:head# 2024-04-25T07:16:06.412 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.454 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a504a533:::existing_90:head# 2024-04-25T07:16:06.454 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.495 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a691337e:::existing_61:head# 2024-04-25T07:16:06.495 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.545 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-04-25T07:16:06.546 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.587 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a9fde86e:::existing_18:head# 2024-04-25T07:16:06.587 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.629 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aa309faa:::existing_68:head# 2024-04-25T07:16:06.629 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.671 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-04-25T07:16:06.671 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.738 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b279b1fe:::existing_10:head# 2024-04-25T07:16:06.738 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.779 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b343e277:::existing_88:head# 2024-04-25T07:16:06.780 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.822 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b7130e90:::existing_80:head# 2024-04-25T07:16:06.822 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.905 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc6f7309:::existing_70:head# 2024-04-25T07:16:06.905 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:06.964 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:be8489b7:::existing_38:head# 2024-04-25T07:16:06.964 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.005 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-04-25T07:16:07.006 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.047 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bebec877:::existing_74:head# 2024-04-25T07:16:07.047 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.089 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c04347d8:::existing_60:head# 2024-04-25T07:16:07.089 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.131 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c156033e:::existing_100:head# 2024-04-25T07:16:07.131 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.173 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-04-25T07:16:07.173 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.214 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-04-25T07:16:07.215 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.256 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-04-25T07:16:07.256 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.348 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-04-25T07:16:07.348 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.432 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ccb3865b:::existing_73:head# 2024-04-25T07:16:07.432 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.507 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cd84e19e:::existing_40:head# 2024-04-25T07:16:07.508 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.549 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-04-25T07:16:07.549 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.591 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d41a25c9:::existing_22:head# 2024-04-25T07:16:07.591 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.633 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d7f088da:::existing_91:head# 2024-04-25T07:16:07.633 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.675 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:16:07.675 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.716 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-04-25T07:16:07.716 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.758 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dd677bd3:::existing_35:head# 2024-04-25T07:16:07.758 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.800 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-04-25T07:16:07.800 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.842 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e0e22077:::existing_21:head# 2024-04-25T07:16:07.842 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.883 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e255b710:::existing_55:head# 2024-04-25T07:16:07.884 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.926 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:eff1d5db:::existing_14:head# 2024-04-25T07:16:07.926 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.959 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f1b156bf:::existing_23:head# 2024-04-25T07:16:07.959 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:07.992 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f5ce4199:::existing_34:head# 2024-04-25T07:16:07.993 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:08.026 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f6809456:::existing_11:head# 2024-04-25T07:16:08.026 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:08.059 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f699a2a9:::existing_67:head# 2024-04-25T07:16:08.060 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:08.093 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f9baac43:::existing_25:head# 2024-04-25T07:16:08.093 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:08.126 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fb76688e:::existing_51:head# 2024-04-25T07:16:08.127 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:08.160 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fbb92eec:::existing_69:head# 2024-04-25T07:16:08.160 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:16:08.193 INFO:tasks.workunit.client.0.smithi102.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-04-25T07:16:08.235 INFO:tasks.workunit.client.0.smithi102.stdout:Import successful 2024-04-25T07:16:08.774 INFO:tasks.workunit.client.0.smithi102.stdout:reviving divergent 1 2024-04-25T07:16:08.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-04-25T07:16:08.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:09.171 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:09.171 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 4 4 0 0 62050 0 0 11 90 11 active+recovering+degraded 2024-04-25T07:15:59.093298+0000 41'101 47:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:16:09.171 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:16:09.171 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:09.171 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:16:09.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-04-25T07:16:09.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:16:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:16:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:16:09.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:16:09.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:16:09.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:16:09.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:16:09.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:16:09.191 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:16:09.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:16:09.192 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:16:09.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:16:09.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:16:09.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:16:09.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:16:09.222 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:09.227+0000 7f39568f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:09.224 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:09.227+0000 7f39568f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:09.226 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:09.227+0000 7f39568f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:09.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:16:09.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:16:09.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:16:09.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:16:09.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:16:09.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:09.595 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:16:09.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:16:09.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:09.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:16:09.784 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:09.787+0000 7f39568f77c0 -1 Falling back to public interface 2024-04-25T07:16:10.020 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/879082079,v1:127.0.0.1:6819/879082079] [v2:127.0.0.1:6820/879082079,v1:127.0.0.1:6821/879082079] exists,up 0fa44109-b9ae-4d0d-9eb0-a7b7d85f02fe 2024-04-25T07:16:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:16:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:16:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:16:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-04-25T07:16:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:16:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:16:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:16:10.022 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:16:10.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:16:10.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:10.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:16:10.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:10.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:16:10.436 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/879082079,v1:127.0.0.1:6819/879082079] [v2:127.0.0.1:6820/879082079,v1:127.0.0.1:6821/879082079] exists,up 0fa44109-b9ae-4d0d-9eb0-a7b7d85f02fe 2024-04-25T07:16:10.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:16:10.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:16:10.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:16:10.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-04-25T07:16:12.764 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:12.767+0000 7f39568f77c0 -1 osd.1 47 log_to_monitors true 2024-04-25T07:16:30.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-04-25T07:16:30.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:16:30.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:16:30.439 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:16:30.439 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:30.439 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:16:30.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.1.asok 2024-04-25T07:16:30.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:16:30.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T07:16:30.554 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:16:30.555 INFO:tasks.workunit.client.0.smithi102.stdout: "ops": [], 2024-04-25T07:16:30.555 INFO:tasks.workunit.client.0.smithi102.stdout: "num_ops": 0 2024-04-25T07:16:30.555 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:16:30.567 INFO:tasks.workunit.client.0.smithi102.stdout:allowing recovery 2024-04-25T07:16:30.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-04-25T07:16:30.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:30.974 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:30.974 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-04-25T07:16:15.375496+0000 47'102 52:156 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:41:27.453714+0000 0 0 2024-04-25T07:16:30.974 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:16:30.975 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:30.975 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:16:30.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:16:30.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:16:31.143 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:16:31.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:16:31.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:16:31.323 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:16:31.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:16:31.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:16:31.490 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:16:31.504 INFO:tasks.workunit.client.0.smithi102.stdout:reading divergent objects 2024-04-25T07:16:31.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-04-25T07:16:31.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:31.884 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:31.884 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-04-25T07:16:15.375496+0000 47'102 52:156 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:13:31.511225+0000 0'0 2024-04-25T07:13:31.511225+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:41:27.453714+0000 0 0 2024-04-25T07:16:31.884 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:16:31.884 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:31.884 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:16:31.900 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-04-25T07:16:31.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-04-25T07:16:31.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:16:31.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:16:31.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:16:31.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:16:31.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:16:31.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:16:32.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:16:32.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:16:32.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-04-25T07:16:32.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:16:32.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-04-25T07:16:32.076 INFO:tasks.workunit.client.0.smithi102.stderr:error getting test/newobject_1: (2) No such file or directory 2024-04-25T07:16:32.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:16:32.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-04-25T07:16:32.108 INFO:tasks.workunit.client.0.smithi102.stderr:error getting test/newobject_2: (2) No such file or directory 2024-04-25T07:16:32.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-04-25T07:16:32.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:16:32.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:16:32.116 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4635.0:1 2024-04-25T07:14:08.296337+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:16:32.116 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: keeping 29'104 (29'4) delete 1:70b3bf12:::existing_4:head by client.4635.0:1 2024-04-25T07:14:08.296337+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:16:32.116 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:16:32.116 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-04-25T07:16:32.116 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: more recent entry found: 41'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4756.0:1 2024-04-25T07:15:11.892175+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-04-25T07:16:32.116 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4630.0:1 2024-04-25T07:14:08.291149+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:16:32.116 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4630.0:1 2024-04-25T07:14:08.291149+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:16:32.116 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:16:32.116 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-04-25T07:14:08.301344+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: keeping 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-04-25T07:14:08.301344+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'106 (29'3) delete 1:8881e6d8:::existing_3:head by client.4639.0:1 2024-04-25T07:14:08.301344+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-04-25T07:16:32.117 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4627.0:1 2024-04-25T07:14:08.279670+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4627.0:1 2024-04-25T07:14:08.279670+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4627.0:1 2024-04-25T07:14:08.279670+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-04-25T07:14:08.299803+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:16:32.118 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: keeping 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-04-25T07:14:08.299803+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:16:32.119 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:16:32.119 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-04-25T07:16:32.119 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-04-25T07:16:32.119 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:16:32.119 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-04-25T07:16:32.119 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'105 (29'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-04-25T07:14:08.299803+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:16:32.119 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-04-25T07:16:32.119 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-04-25T07:16:32.119 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4626.0:1 2024-04-25T07:14:08.279207+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:16:32.119 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4626.0:1 2024-04-25T07:14:08.279207+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:16:32.120 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:16:32.120 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-04-25T07:16:32.120 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-04-25T07:16:32.120 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:15:58.343+0000 7f97dc046640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-04-25T07:16:32.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:16:32.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:16:32.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:16:32.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:16:32.128 INFO:tasks.workunit.client.0.smithi102.stdout:success 2024-04-25T07:16:32.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-04-25T07:16:32.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.26156.out 2024-04-25T07:16:32.129 INFO:tasks.workunit.client.0.smithi102.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.26156.out': No such file or directory 2024-04-25T07:16:32.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-04-25T07:16:32.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:16:32.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:16:32.766 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T07:16:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-04-25T07:16:32.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:16:32.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:16:32.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:16:32.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:16:32.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:16:33.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:16:33.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:16:33.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:16:33.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:16:33.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:16:33.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:16:33.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:16:33.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:16:33.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:16:33.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:16:33.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:16:33.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:16:33.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:16:33.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:16:33.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:16:33.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:16:33.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:16:33.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:16:33.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:16:33.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:16:33.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:16:33.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:16:33.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:16:33.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:16:33.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:16:33.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:33.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:16:33.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26156 2024-04-25T07:16:33.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:16:33.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:16:33.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:16:33.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:16:33.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:16:33.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:16:33.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:16:33.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:16:33.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:16:33.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:16:33.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:16:33.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:16:33.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:16:33.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:16:33.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:16:33.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:16:33.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:16:33.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:33.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:16:33.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26156 2024-04-25T07:16:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:16:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:16:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:16:33.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:16:33.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:33.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:16:33.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26156 2024-04-25T07:16:33.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:16:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:16:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:16:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:16:33.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-04-25T07:16:33.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-04-25T07:16:33.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-04-25T07:16:33.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-04-25T07:16:33.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-04-25T07:16:33.147 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-04-25T07:16:33.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-04-25T07:16:33.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-04-25T07:16:33.148 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:16:33.148 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:16:33.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-04-25T07:16:33.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:16:33.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:16:33.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:16:33.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:16:33.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:16:33.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:16:33.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:16:33.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:16:33.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:16:33.636 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:16:33.636 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:33.636 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:16:33.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:16:33.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:16:33.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:16:33.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:16:33.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:16:33.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:16:33.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:16:33.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:16:33.849 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:16:33.849 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:16:33.849 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:16:33.850 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:16:33.850 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:33.850 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:16:33.850 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-mon.a.asok 2024-04-25T07:16:33.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:16:33.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26156/ceph-mon.a.asok config get fsid 2024-04-25T07:16:33.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:16:33.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:16:33.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:16:33.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:16:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:16:33.967 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:16:33.968 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:16:33.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:16:33.968 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:16:33.968 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:33.968 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:16:33.968 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-mon.a.asok 2024-04-25T07:16:33.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:16:33.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26156/ceph-mon.a.asok config get mon_host 2024-04-25T07:16:34.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-04-25T07:16:34.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:16:34.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:16:34.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:16:34.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:16:34.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:16:34.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:16:34.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:16:34.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:16:34.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:16:34.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:16:34.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:34.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:16:34.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:16:34.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:16:34.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:16:35.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:16:35.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:16:35.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:16:35.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3d1dc328-65a0-4947-a470-7ad55360f6cb 2024-04-25T07:16:35.039 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 3d1dc328-65a0-4947-a470-7ad55360f6cb 2024-04-25T07:16:35.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3d1dc328-65a0-4947-a470-7ad55360f6cb' 2024-04-25T07:16:35.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:16:35.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBTAypmwCrcAxAAHhdAdYJsT4fnTW3KxE+CCg== 2024-04-25T07:16:35.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBTAypmwCrcAxAAHhdAdYJsT4fnTW3KxE+CCg=="}' 2024-04-25T07:16:35.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3d1dc328-65a0-4947-a470-7ad55360f6cb -i td/divergent-priors/0/new.json 2024-04-25T07:16:35.391 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:16:35.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:16:35.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBTAypmwCrcAxAAHhdAdYJsT4fnTW3KxE+CCg== --osd-uuid 3d1dc328-65a0-4947-a470-7ad55360f6cb 2024-04-25T07:16:35.450 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:35.455+0000 7fae990537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:35.460 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:35.463+0000 7fae990537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:35.461 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:35.463+0000 7fae990537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:35.463+0000 7fae990537c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:16:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:35.463+0000 7fae990537c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:16:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:35.463+0000 7fae990537c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:16:35.500 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:35.503+0000 7fae990537c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:16:41.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:16:41.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:16:41.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:16:41.801 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:16:41.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:16:42.375 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:16:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:16:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:16:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:16:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:16:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:16:42.418 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:42.423+0000 7f0481a217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:42.450 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:42.455+0000 7f0481a217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:42.452 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:42.455+0000 7f0481a217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:42.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:16:42.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:16:42.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:16:42.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:16:42.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:16:42.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:42.815 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:16:42.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:16:42.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:42.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:43.023 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:43.027+0000 7f0481a217c0 -1 Falling back to public interface 2024-04-25T07:16:43.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:44.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:44.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:44.263 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:16:44.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:16:44.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:44.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:44.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:45.408 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:16:45.411+0000 7f0481a217c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:16:45.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:45.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:45.704 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:16:45.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:16:45.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:45.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:46.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:47.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:47.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:47.262 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:16:47.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:16:47.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:47.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:47.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:48.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:48.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:48.694 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:16:48.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:16:48.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:48.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:49.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:50.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:50.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:50.263 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:16:50.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:16:50.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:50.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:50.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:51.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:51.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:51.709 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:16:51.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:16:51.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:51.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:52.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:53.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:53.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:53.321 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:16:53.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:16:53.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:53.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:53.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:54.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:54.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:54.802 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:16:54.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:16:54.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:54.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:55.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:56.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:56.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:56.313 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:16:56.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:16:56.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:56.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:57.745 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:16:57.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:16:57.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:57.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:58.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:16:59.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:16:59.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:59.265 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:16:59.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:16:59.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:59.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:00.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:01.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:01.719 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:17:01.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:17:01.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:01.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:02.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:03.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:03.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:03.951 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:17:03.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:17:03.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:03.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:04.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:05.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:05.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:05.705 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:17:05.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:17:05.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:05.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:07.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:08.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:08.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:08.022 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:17:08.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:17:08.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:08.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:08.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:09.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:09.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:09.669 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:17:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:17:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:10.191 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/640311997,v1:127.0.0.1:6803/640311997] [v2:127.0.0.1:6804/640311997,v1:127.0.0.1:6805/640311997] exists,up 3d1dc328-65a0-4947-a470-7ad55360f6cb 2024-04-25T07:17:10.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:17:10.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:17:10.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:17:10.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:17:10.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:17:10.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:17:10.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:17:10.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:17:10.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:17:10.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:17:10.195 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:17:10.195 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:10.195 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:17:10.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:17:10.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:17:10.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:17:10.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:17:10.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:17:10.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:17:10.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:17:10.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:17:10.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:17:10.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:17:10.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:17:10.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:17:10.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:17:10.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:17:10.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:17:10.197 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 df369e44-fe63-4f08-84e4-07f167676b56 2024-04-25T07:17:10.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=df369e44-fe63-4f08-84e4-07f167676b56 2024-04-25T07:17:10.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 df369e44-fe63-4f08-84e4-07f167676b56' 2024-04-25T07:17:10.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:17:10.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB2AypmZjzeDRAA6n9GL3HqiCMY+r2TAFn96A== 2024-04-25T07:17:10.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB2AypmZjzeDRAA6n9GL3HqiCMY+r2TAFn96A=="}' 2024-04-25T07:17:10.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new df369e44-fe63-4f08-84e4-07f167676b56 -i td/divergent-priors/1/new.json 2024-04-25T07:17:10.882 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:17:10.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:17:10.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB2AypmZjzeDRAA6n9GL3HqiCMY+r2TAFn96A== --osd-uuid df369e44-fe63-4f08-84e4-07f167676b56 2024-04-25T07:17:10.945 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:10.946+0000 7f02ccb097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:10.966 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:10.970+0000 7f02ccb097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:10.967 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:10.970+0000 7f02ccb097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:10.968 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:10.970+0000 7f02ccb097c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:17:10.968 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:10.970+0000 7f02ccb097c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:17:10.968 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:10.970+0000 7f02ccb097c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:17:11.058 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:11.062+0000 7f02ccb097c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:17:17.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:17:17.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:17:17.577 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:17:17.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:17:17.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:17:18.148 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:17:18.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:17:18.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:17:18.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:17:18.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:17:18.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:17:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:18.190+0000 7fa33824c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:18.222 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:18.226+0000 7fa33824c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:18.223 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:18.226+0000 7fa33824c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:18.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:17:18.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:17:18.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:17:18.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:17:18.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:17:18.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:18.610 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:17:18.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:17:18.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:18.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:18.807 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:18.810+0000 7fa33824c7c0 -1 Falling back to public interface 2024-04-25T07:17:19.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:20.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:20.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:20.031 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:17:20.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:17:20.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:20.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:20.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:21.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:21.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:21.493 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:17:21.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:17:21.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:21.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:21.633 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:21.634+0000 7fa33824c7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:17:22.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:23.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:23.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:23.035 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:17:23.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:17:23.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:23.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:24.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:24.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:24.505 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:17:24.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:17:24.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:24.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:25.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:26.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:26.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:26.031 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:17:26.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:17:26.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:26.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:26.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:27.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:27.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:27.661 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:17:27.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:17:27.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:27.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:28.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:29.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:29.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:29.106 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:17:29.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:17:29.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:29.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:29.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:30.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:30.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:30.671 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:17:30.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:17:30.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:30.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:31.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:32.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:32.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:32.202 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:17:32.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:17:32.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:32.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:32.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:33.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:33.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:33.920 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:17:33.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:17:33.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:33.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:34.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:35.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:35.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:35.511 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:17:35.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:17:35.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:35.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:35.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:36.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:36.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:36.986 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:17:36.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:17:36.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:36.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:37.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:38.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:38.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:38.439 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:17:38.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:17:38.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:38.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:39.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:40.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:40.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:40.246 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:17:40.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:17:40.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:40.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:41.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:42.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:42.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:42.395 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:17:42.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:17:42.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:42.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:43.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:44.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:44.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:44.202 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:17:44.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:17:44.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:44.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:45.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:46.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:46.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:46.028 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:17:46.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:17:46.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:46.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:46.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:47.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:47.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:47.894 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:17:47.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:17:47.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:47.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:17:48.357 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2000621781,v1:127.0.0.1:6811/2000621781] [v2:127.0.0.1:6812/2000621781,v1:127.0.0.1:6813/2000621781] exists,up df369e44-fe63-4f08-84e4-07f167676b56 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:17:48.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:17:48.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:17:48.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:17:48.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:17:48.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:17:48.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:17:48.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:17:48.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:17:48.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:17:48.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:17:48.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:17:48.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:17:48.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:17:48.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:17:48.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:17:48.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:17:48.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dcba27a8-55a9-40c9-9cc7-d09a609befd5 2024-04-25T07:17:48.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 dcba27a8-55a9-40c9-9cc7-d09a609befd5' 2024-04-25T07:17:48.364 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 dcba27a8-55a9-40c9-9cc7-d09a609befd5 2024-04-25T07:17:48.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:17:48.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCcAypm6Q5NFxAAExrthhp6Hw7W7mDW/9rQpw== 2024-04-25T07:17:48.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCcAypm6Q5NFxAAExrthhp6Hw7W7mDW/9rQpw=="}' 2024-04-25T07:17:48.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dcba27a8-55a9-40c9-9cc7-d09a609befd5 -i td/divergent-priors/2/new.json 2024-04-25T07:17:49.012 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:17:49.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:17:49.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCcAypm6Q5NFxAAExrthhp6Hw7W7mDW/9rQpw== --osd-uuid dcba27a8-55a9-40c9-9cc7-d09a609befd5 2024-04-25T07:17:49.070 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:49.074+0000 7f777923b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:49.097 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:49.098+0000 7f777923b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:49.098 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:49.102+0000 7f777923b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:49.099 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:49.102+0000 7f777923b7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:17:49.099 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:49.102+0000 7f777923b7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:17:49.099 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:49.102+0000 7f777923b7c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:17:49.136 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:49.138+0000 7f777923b7c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:17:55.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:17:55.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:17:55.784 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:17:55.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:17:55.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:17:56.403 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:17:56.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:17:56.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:17:56.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:17:56.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:17:56.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:17:56.443 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:56.446+0000 7ff634ebd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:56.478 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:56.482+0000 7ff634ebd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:56.479 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:56.482+0000 7ff634ebd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:56.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:17:56.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:17:56.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:17:56.866 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:17:56.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:17:56.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:17:56.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:56.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:17:56.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:17:56.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:57.043 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:57.046+0000 7ff634ebd7c0 -1 Falling back to public interface 2024-04-25T07:17:57.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:58.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:58.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:58.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:17:58.298 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:17:58.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:58.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:17:58.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:59.693 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:17:59.694+0000 7ff634ebd7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:17:59.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:59.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:59.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:17:59.708 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:17:59.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:59.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:00.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:01.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:01.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:01.145 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:18:01.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:18:01.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:01.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:01.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:02.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:02.693 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:18:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:18:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:03.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:04.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:04.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:04.200 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:18:04.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:18:04.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:04.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:04.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:05.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:05.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:05.786 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:18:05.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:18:05.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:05.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:06.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:07.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:07.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:07.470 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:18:07.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:18:07.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:07.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:08.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:09.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:09.046 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:18:09.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:09.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:18:09.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:09.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:10.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:10.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:10.473 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:18:10.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:18:10.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:10.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:10.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:11.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:11.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:11.934 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:18:11.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:18:11.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:11.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:13.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:13.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:13.534 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:18:13.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:18:13.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:13.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:14.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:15.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:15.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:15.270 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:18:15.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:18:15.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:15.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:15.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:16.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:16.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:16.855 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:18:16.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:18:16.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:16.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:17.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:18.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:18.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:18.853 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:18:18.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:18:18.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:18.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:19.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:20.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:20.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:20.694 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:18:20.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:18:20.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:20.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:21.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:22.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:22.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:22.414 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:18:22.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:18:22.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:22.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:23.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:24.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:24.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:24.612 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:18:24.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:18:24.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:24.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:25.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:26.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:26.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:26.096 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:18:26.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:18:26.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:26.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:26.533 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/240408323,v1:127.0.0.1:6819/240408323] [v2:127.0.0.1:6820/240408323,v1:127.0.0.1:6821/240408323] exists,up dcba27a8-55a9-40c9-9cc7-d09a609befd5 2024-04-25T07:18:26.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:18:26.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:18:26.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:18:26.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-04-25T07:18:27.226 INFO:tasks.workunit.client.0.smithi102.stderr:noout is set 2024-04-25T07:18:27.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-04-25T07:18:28.163 INFO:tasks.workunit.client.0.smithi102.stderr:noin is set 2024-04-25T07:18:28.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-04-25T07:18:29.017 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T07:18:29.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-04-25T07:18:29.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T07:18:29.728 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T07:18:29.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:18:30.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-04-25T07:18:31.545 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 3 2024-04-25T07:18:31.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-04-25T07:18:32.366 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 min_size to 2 2024-04-25T07:18:32.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-04-25T07:18:33.102 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 pg_autoscale_mode to on 2024-04-25T07:18:33.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-04-25T07:18:33.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-04-25T07:18:33.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1714029513 2024-04-25T07:18:33.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-04-25T07:18:33.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-04-25T07:18:33.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-04-25T07:18:33.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:18:33.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:18:33.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:18:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:18:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:18:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:18:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:33.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:18:33.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836502 2024-04-25T07:18:33.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836502 2024-04-25T07:18:33.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502' 2024-04-25T07:18:33.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:33.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:18:33.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672976 2024-04-25T07:18:33.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672976 2024-04-25T07:18:33.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672976' 2024-04-25T07:18:33.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:33.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:18:34.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509448 2024-04-25T07:18:34.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509448 2024-04-25T07:18:34.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672976 2-64424509448' 2024-04-25T07:18:34.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:34.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836502 2024-04-25T07:18:34.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:34.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:18:34.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-04-25T07:18:34.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:34.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836502 2024-04-25T07:18:34.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-04-25T07:18:34.099 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836502 2024-04-25T07:18:34.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:18:34.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836500 -lt 21474836502 2024-04-25T07:18:34.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:18:35.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:18:35.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:18:35.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836502 -lt 21474836502 2024-04-25T07:18:35.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:35.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:35.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672976 2024-04-25T07:18:35.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:18:35.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672976 2024-04-25T07:18:35.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:35.958 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672976 2024-04-25T07:18:35.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672976 2024-04-25T07:18:35.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-04-25T07:18:35.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:18:36.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672976 2024-04-25T07:18:36.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:36.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509448 2024-04-25T07:18:36.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:36.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:18:36.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509448 2024-04-25T07:18:36.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:36.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509448 2024-04-25T07:18:36.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-04-25T07:18:36.375 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509448 2024-04-25T07:18:36.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:18:36.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509448 -lt 64424509448 2024-04-25T07:18:36.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-04-25T07:18:36.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:18:36.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:18:36.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:18:36.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:18:36.774 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:18:36.774 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:18:36.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:18:36.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:18:36.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:18:36.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:18:36.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:18:36.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:18:36.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:18:36.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:18:36.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:18:37.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:18:37.305 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:18:37.305 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:18:37.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:18:37.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:37.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:18:37.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836505 2024-04-25T07:18:37.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836505 2024-04-25T07:18:37.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505' 2024-04-25T07:18:37.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:37.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:18:37.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-04-25T07:18:37.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-04-25T07:18:37.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672978' 2024-04-25T07:18:37.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:37.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:18:37.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509451 2024-04-25T07:18:37.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509451 2024-04-25T07:18:37.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672978 2-64424509451' 2024-04-25T07:18:37.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:37.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836505 2024-04-25T07:18:37.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:37.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:18:37.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836505 2024-04-25T07:18:37.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:37.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836505 2024-04-25T07:18:37.802 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836505 2024-04-25T07:18:37.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-04-25T07:18:37.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:18:38.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836505 2024-04-25T07:18:38.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:38.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-04-25T07:18:38.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:38.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:18:38.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-04-25T07:18:38.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:38.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-04-25T07:18:38.252 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672978 2024-04-25T07:18:38.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-25T07:18:38.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:18:38.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672978 2024-04-25T07:18:38.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:18:39.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:18:39.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:18:40.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672978 2024-04-25T07:18:40.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:40.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509451 2024-04-25T07:18:40.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:18:40.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509451 2024-04-25T07:18:40.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:40.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509451 2024-04-25T07:18:40.170 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509451 2024-04-25T07:18:40.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-04-25T07:18:40.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:18:40.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509451 -lt 64424509451 2024-04-25T07:18:40.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:18:40.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:18:40.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:18:41.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:18:41.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:18:41.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:18:41.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:18:41.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:18:41.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:18:41.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:18:41.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:18:41.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:18:41.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:18:41.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:18:41.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:18:42.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:18:42.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:18:42.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:18:42.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-04-25T07:18:42.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-04-25T07:18:42.627 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:18:42.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=-1 2024-04-25T07:18:42.644 INFO:tasks.workunit.client.0.smithi102.stdout:primary and soon to be divergent is -1 2024-04-25T07:18:42.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is -1' 2024-04-25T07:18:42.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:18:43.179 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:18:43.179 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.179 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.179 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.179 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.179 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.179 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:30.801829+0000 0'0 31:23 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:37:39.229424+0000 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.180 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 0 0 0 0 0 0 0 0 0 0 0 unknown 2024-04-25T07:18:41.635650+0000 0'0 0:0 [] -1 [] -1 0'0 2024-04-25T07:18:41.635650+0000 0'0 2024-04-25T07:18:41.635650+0000 0 0 -- 0 0 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:18:43.181 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:18:43.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-04-25T07:18:43.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1714029523 2024-04-25T07:18:43.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1714029523 - 1714029513 2024-04-25T07:18:43.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=10 2024-04-25T07:18:43.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 10 -gt 300 ']' 2024-04-25T07:18:43.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-04-25T07:18:43.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-04-25T07:18:43.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:18:43.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:18:43.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:18:43.619 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:18:43.619 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:18:43.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:18:43.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:43.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:18:43.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836508 2024-04-25T07:18:43.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836508 2024-04-25T07:18:43.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508' 2024-04-25T07:18:43.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:43.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:18:44.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672982 2024-04-25T07:18:44.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672982 2024-04-25T07:18:44.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672982' 2024-04-25T07:18:44.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:44.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:18:44.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509454 2024-04-25T07:18:44.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509454 2024-04-25T07:18:44.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672982 2-64424509454' 2024-04-25T07:18:44.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:44.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836508 2024-04-25T07:18:44.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:44.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:18:44.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-04-25T07:18:44.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:44.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836508 2024-04-25T07:18:44.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-04-25T07:18:44.220 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836508 2024-04-25T07:18:44.220 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:18:44.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836508 2024-04-25T07:18:44.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:18:45.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:18:45.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:18:46.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836508 2024-04-25T07:18:46.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:46.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672982 2024-04-25T07:18:46.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:46.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:18:46.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-04-25T07:18:46.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:46.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672982 2024-04-25T07:18:46.400 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672982 2024-04-25T07:18:46.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-04-25T07:18:46.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:18:47.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672982 2024-04-25T07:18:47.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:47.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:47.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509454 2024-04-25T07:18:47.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:18:47.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509454 2024-04-25T07:18:47.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:47.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509454 2024-04-25T07:18:47.018 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509454 2024-04-25T07:18:47.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-04-25T07:18:47.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:18:47.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509454 2024-04-25T07:18:47.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-04-25T07:18:47.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:18:47.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:18:47.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:18:47.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:18:47.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:18:47.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:18:47.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:18:47.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:18:47.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:18:47.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:18:47.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:18:47.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:18:47.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:18:47.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:18:47.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:18:48.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:18:48.197 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:18:48.197 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:18:48.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:18:48.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:48.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:18:48.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-04-25T07:18:48.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-04-25T07:18:48.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-04-25T07:18:48.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:48.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:18:48.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-04-25T07:18:48.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-04-25T07:18:48.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672985' 2024-04-25T07:18:48.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:18:48.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:18:48.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509457 2024-04-25T07:18:48.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509457 2024-04-25T07:18:48.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672985 2-64424509457' 2024-04-25T07:18:48.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:48.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-04-25T07:18:48.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:48.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:18:48.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-04-25T07:18:48.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:48.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-04-25T07:18:48.698 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836511 2024-04-25T07:18:48.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-04-25T07:18:48.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:18:49.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836511 2024-04-25T07:18:49.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:18:50.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:18:50.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:18:50.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836511 2024-04-25T07:18:50.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:50.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-04-25T07:18:50.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:50.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:18:50.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-04-25T07:18:50.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:50.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-04-25T07:18:50.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-25T07:18:50.537 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672985 2024-04-25T07:18:50.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:18:51.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-04-25T07:18:51.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:18:51.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509457 2024-04-25T07:18:51.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:18:51.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:18:51.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509457 2024-04-25T07:18:51.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:18:51.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509457 2024-04-25T07:18:51.053 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509457 2024-04-25T07:18:51.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-04-25T07:18:51.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:18:51.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509457 2024-04-25T07:18:51.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:18:51.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:18:51.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:18:52.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:18:52.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:18:52.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:18:52.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:18:52.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:18:52.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:18:52.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:18:52.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:18:52.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-04-25T07:18:52.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:18:52.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:18:52.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:18:53.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-04-25T07:18:53.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:18:53.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:18:53.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-04-25T07:18:53.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-04-25T07:18:53.713 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:18:53.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=0 2024-04-25T07:18:53.731 INFO:tasks.workunit.client.0.smithi102.stdout:primary and soon to be divergent is 0 2024-04-25T07:18:53.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 0' 2024-04-25T07:18:53.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:18:54.118 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:18:54.118 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:49.603907+0000 0'0 37:47 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:17:44.581101+0000 0 0 2024-04-25T07:18:54.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.720794+0000 0'0 37:51 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:09:14.685191+0000 0 0 2024-04-25T07:18:54.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:44.730861+0000 0'0 37:51 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:28:16.294094+0000 0 0 2024-04-25T07:18:54.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:51.728026+0000 0'0 37:51 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:04:51.414488+0000 0 0 2024-04-25T07:18:54.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.112079+0000 0'0 37:47 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:54.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.112319+0000 0'0 37:47 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:54.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.112166+0000 0'0 37:47 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:54.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.114140+0000 0'0 37:47 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:54.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:42.838124+0000 0'0 37:43 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:54.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:49.854281+0000 0'0 37:48 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:32:12.748825+0000 0 0 2024-04-25T07:18:54.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:50.740506+0000 0'0 37:51 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:12:05.054924+0000 0 0 2024-04-25T07:18:54.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.111553+0000 0'0 37:47 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:54.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.514859+0000 0'0 37:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:54:30.505712+0000 0 0 2024-04-25T07:18:54.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:49.751722+0000 0'0 37:51 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:43:50.607718+0000 0 0 2024-04-25T07:18:54.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.815527+0000 0'0 37:48 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:52:56.067478+0000 0 0 2024-04-25T07:18:54.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.113947+0000 0'0 37:39 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:54.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:43.669542+0000 0'0 37:51 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:32:10.336941+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:47.844714+0000 0'0 37:48 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:21:35.075870+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.112908+0000 0'0 37:47 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:48.891102+0000 0'0 37:48 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:54:44.155493+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:48.591032+0000 0'0 37:47 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:29:27.576830+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:47.552772+0000 0'0 37:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:59:09.784006+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.113345+0000 0'0 37:39 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:45.842787+0000 0'0 37:51 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:17:10.472671+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:50.875188+0000 0'0 37:48 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:33:34.607451+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:52.731903+0000 0'0 37:51 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:14:36.657130+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:48.724703+0000 0'0 37:51 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:37:52.999573+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.112739+0000 0'0 37:39 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:45.841071+0000 0'0 37:48 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:57:25.545147+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:47.700445+0000 0'0 37:51 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:15:42.736676+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:46.110947+0000 0'0 37:47 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:29.457836+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 queued for scrub 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:18:50.605832+0000 0'0 37:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:49:02.053366+0000 0 0 2024-04-25T07:18:54.121 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:18:54.122 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:18:54.122 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:18:54.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-04-25T07:18:54.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1714029534 2024-04-25T07:18:54.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1714029534 - 1714029513 2024-04-25T07:18:54.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=21 2024-04-25T07:18:54.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 21 -gt 300 ']' 2024-04-25T07:18:54.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 0 -le -1 ']' 2024-04-25T07:18:54.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-04-25T07:18:54.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:18:54.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 0 ']' 2024-04-25T07:18:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-04-25T07:18:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:18:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 0 ']' 2024-04-25T07:18:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 1' 2024-04-25T07:18:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:18:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 0 ']' 2024-04-25T07:18:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 1 2' 2024-04-25T07:18:54.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-04-25T07:18:54.139 INFO:tasks.workunit.client.0.smithi102.stdout:writing initial objects 2024-04-25T07:18:54.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-04-25T07:18:54.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:54.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-04-25T07:18:54.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:54.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-04-25T07:18:54.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:54.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-04-25T07:18:54.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:54.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-04-25T07:18:55.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:55.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-04-25T07:18:55.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:55.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-04-25T07:18:55.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:55.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-04-25T07:18:55.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:55.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-04-25T07:18:55.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:55.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-04-25T07:18:56.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:56.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-04-25T07:18:56.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:56.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-04-25T07:18:56.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:56.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-04-25T07:18:56.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:56.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-04-25T07:18:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:57.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-04-25T07:18:57.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:57.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-04-25T07:18:57.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:57.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-04-25T07:18:57.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:57.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-04-25T07:18:57.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:57.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-04-25T07:18:58.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:58.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-04-25T07:18:58.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:58.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-04-25T07:18:58.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:58.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-04-25T07:18:58.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:58.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-04-25T07:18:58.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:58.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-04-25T07:18:59.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:59.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-04-25T07:18:59.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:59.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-04-25T07:18:59.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:59.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-04-25T07:18:59.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:59.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-04-25T07:18:59.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:18:59.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-04-25T07:19:00.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:00.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-04-25T07:19:00.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:00.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-04-25T07:19:00.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:00.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-04-25T07:19:00.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:00.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-04-25T07:19:00.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:00.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-04-25T07:19:01.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:01.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-04-25T07:19:01.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:01.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-04-25T07:19:01.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:01.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-04-25T07:19:01.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:01.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-04-25T07:19:01.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:01.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-04-25T07:19:02.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:02.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-04-25T07:19:02.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:02.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-04-25T07:19:02.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:02.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-04-25T07:19:02.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:02.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-04-25T07:19:02.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:02.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-04-25T07:19:02.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:02.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-04-25T07:19:03.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:03.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-04-25T07:19:03.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:03.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-04-25T07:19:03.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:03.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-04-25T07:19:03.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:03.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-04-25T07:19:03.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:03.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-04-25T07:19:03.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:03.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-04-25T07:19:04.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:04.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-04-25T07:19:04.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:04.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-04-25T07:19:04.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:04.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-04-25T07:19:04.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:04.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-04-25T07:19:04.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:04.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-04-25T07:19:05.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:05.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-04-25T07:19:05.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:05.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-04-25T07:19:05.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:05.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-04-25T07:19:05.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:05.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-04-25T07:19:05.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:05.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-04-25T07:19:05.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:05.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-04-25T07:19:06.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:06.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-04-25T07:19:06.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:06.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-04-25T07:19:06.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:06.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-04-25T07:19:06.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:06.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-04-25T07:19:06.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:06.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-04-25T07:19:07.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:07.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-04-25T07:19:07.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:07.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-04-25T07:19:07.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:07.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-04-25T07:19:07.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:07.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-04-25T07:19:07.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:07.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-04-25T07:19:07.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:07.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-04-25T07:19:08.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:08.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-04-25T07:19:08.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:08.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-04-25T07:19:08.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:08.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-04-25T07:19:08.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:08.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-04-25T07:19:08.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:08.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-04-25T07:19:09.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:09.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-04-25T07:19:09.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:09.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-04-25T07:19:09.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:09.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-04-25T07:19:09.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:09.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-04-25T07:19:09.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:09.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-04-25T07:19:09.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:09.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-04-25T07:19:10.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:10.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-04-25T07:19:10.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:10.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-04-25T07:19:10.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:10.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-04-25T07:19:10.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:10.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-04-25T07:19:10.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:10.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-04-25T07:19:10.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:10.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-04-25T07:19:11.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:11.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-04-25T07:19:11.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:11.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-04-25T07:19:11.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:11.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-04-25T07:19:11.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:11.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-04-25T07:19:11.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:11.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-04-25T07:19:12.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:12.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-04-25T07:19:12.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:12.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-04-25T07:19:12.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:12.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-04-25T07:19:12.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:12.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-04-25T07:19:12.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:12.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-04-25T07:19:12.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:12.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-04-25T07:19:13.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:19:13.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-04-25T07:19:13.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:19:13.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:19:13.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:19:13.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:19:13.148 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:19:13.148 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:19:13.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:19:13.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:19:13.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:19:13.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:19:13.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:19:13.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:19:13.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:19:13.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:19:13.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:19:13.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:19:13.720 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:19:13.720 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:19:13.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:19:13.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:13.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:19:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836518 2024-04-25T07:19:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836518 2024-04-25T07:19:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518' 2024-04-25T07:19:13.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:13.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:19:14.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672992 2024-04-25T07:19:14.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672992 2024-04-25T07:19:14.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518 1-42949672992' 2024-04-25T07:19:14.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:14.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:19:14.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509464 2024-04-25T07:19:14.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509464 2024-04-25T07:19:14.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518 1-42949672992 2-64424509464' 2024-04-25T07:19:14.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:14.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836518 2024-04-25T07:19:14.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:14.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:19:14.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836518 2024-04-25T07:19:14.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:14.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836518 2024-04-25T07:19:14.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-04-25T07:19:14.261 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836518 2024-04-25T07:19:14.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:19:14.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836516 -lt 21474836518 2024-04-25T07:19:14.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:19:15.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:19:15.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:19:16.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836518 -lt 21474836518 2024-04-25T07:19:16.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:16.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672992 2024-04-25T07:19:16.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:16.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:19:16.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672992 2024-04-25T07:19:16.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:16.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672992 2024-04-25T07:19:16.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-04-25T07:19:16.090 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672992 2024-04-25T07:19:16.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:19:16.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672992 -lt 42949672992 2024-04-25T07:19:16.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:16.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509464 2024-04-25T07:19:16.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:16.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:19:16.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509464 2024-04-25T07:19:16.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:16.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509464 2024-04-25T07:19:16.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-04-25T07:19:16.546 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509464 2024-04-25T07:19:16.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:19:16.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509464 -lt 64424509464 2024-04-25T07:19:16.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:19:16.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:16.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:17.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:19:17.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:19:17.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:19:17.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:19:17.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:19:17.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:19:17.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:19:17.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:19:17.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-04-25T07:19:17.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:19:17.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:17.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:18.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-04-25T07:19:18.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:19:18.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:19:18.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-04-25T07:19:18.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:19:18.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:19:18.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:19:18.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:19:18.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-04-25T07:19:18.914 INFO:tasks.workunit.client.0.smithi102.stdout:blackholing osds 1 2 2024-04-25T07:19:18.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 1 2' 2024-04-25T07:19:18.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:19:19.296 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:49.603907+0000 37'3 37:50 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:17:44.581101+0000 0 0 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:46.720794+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:09:14.685191+0000 0 0 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:18:44.730861+0000 37'4 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:28:16.294094+0000 0 0 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:51.728026+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:04:51.414488+0000 0 0 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:56.743712+0000 37'3 37:58 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:53:44.268003+0000 1 0 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-04-25T07:18:58.780868+0000 37'5 37:60 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:38:59.269300+0000 2 0 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:57.793345+0000 37'3 37:58 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:23:39.590419+0000 0 0 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:19:00.908033+0000 37'2 37:57 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:49:55.238267+0000 1 0 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:18:53.803820+0000 37'4 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:14:27.294452+0000 0 0 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-04-25T07:18:49.854281+0000 37'6 37:54 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:32:12.748825+0000 0 0 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:18:50.740506+0000 37'2 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:12:05.054924+0000 0 0 2024-04-25T07:19:19.297 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-04-25T07:18:55.832851+0000 37'5 37:60 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:15:58.090184+0000 1 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:18:46.514859+0000 37'1 37:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:54:30.505712+0000 0 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:49.751722+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:43:50.607718+0000 0 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:18:46.815527+0000 37'2 37:50 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:52:56.067478+0000 0 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:18:53.670031+0000 37'2 37:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:25:54.867624+0000 0 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:43.669542+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:32:10.336941+0000 0 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:47.844714+0000 37'3 37:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:21:35.075870+0000 0 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:18:59.847125+0000 37'4 37:59 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:56:12.043910+0000 2 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:18:48.891102+0000 37'4 37:52 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:54:44.155493+0000 0 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:48.591032+0000 37'3 37:50 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:29:27.576830+0000 0 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:47.552772+0000 37'3 37:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:59:09.784006+0000 0 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:18:52.640478+0000 37'1 37:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:18:05.952824+0000 0 0 2024-04-25T07:19:19.298 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:45.842787+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:17:10.472671+0000 0 0 2024-04-25T07:19:19.299 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:50.875188+0000 37'3 37:51 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:33:34.607451+0000 0 0 2024-04-25T07:19:19.299 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:52.731903+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:14:36.657130+0000 0 0 2024-04-25T07:19:19.299 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:18:48.724703+0000 37'4 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:37:52.999573+0000 0 0 2024-04-25T07:19:19.299 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:18:51.601603+0000 37'1 37:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:47:57.800429+0000 0 0 2024-04-25T07:19:19.299 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:45.841071+0000 37'3 37:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:57:25.545147+0000 0 0 2024-04-25T07:19:19.299 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:18:47.700445+0000 37'2 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:15:42.736676+0000 0 0 2024-04-25T07:19:19.299 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:18:54.759791+0000 37'4 37:59 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:06:13.144443+0000 0 0 2024-04-25T07:19:19.299 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-04-25T07:18:50.605832+0000 37'5 37:52 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:49:02.053366+0000 0 0 2024-04-25T07:19:19.299 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:19:19.299 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:19:19.299 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:19:19.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:19:19.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.1 2024-04-25T07:19:19.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:19:19.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:19:19.313 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:19:19.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:19.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:19:19.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.1.asok 2024-04-25T07:19:19.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:19:19.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.1.asok config set objectstore_blackhole 1 2024-04-25T07:19:19.408 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:19:19.408 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:19:19.408 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:19:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:19:19.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-04-25T07:19:19.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:19:19.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:19:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:19:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:19:19.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.2.asok 2024-04-25T07:19:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:19:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:19:19.524 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:19:19.524 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:19:19.524 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:19:19.535 INFO:tasks.workunit.client.0.smithi102.stdout:create new divergent objects 2024-04-25T07:19:19.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-04-25T07:19:19.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-04-25T07:19:19.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:19:19.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:19:19.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-04-25T07:19:19.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-04-25T07:19:19.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-04-25T07:19:19.538 INFO:tasks.workunit.client.0.smithi102.stdout:writing divergent objects 2024-04-25T07:19:19.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-04-25T07:19:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:19:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:19:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-04-25T07:19:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-04-25T07:19:19.542 INFO:tasks.workunit.client.0.smithi102.stdout:remove divergent objects 2024-04-25T07:19:19.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-04-25T07:19:19.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-04-25T07:19:19.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:19:19.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-04-25T07:19:19.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-04-25T07:19:19.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:19:19.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-04-25T07:19:19.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-04-25T07:19:19.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-04-25T07:19:19.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-04-25T07:19:19.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-04-25T07:19:29.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-04-25T07:19:29.554 INFO:tasks.workunit.client.0.smithi102.stderr:rados: no process found 2024-04-25T07:19:29.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-04-25T07:19:29.555 INFO:tasks.workunit.client.0.smithi102.stdout:killing all the osds 2024-04-25T07:19:29.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:19:29.963 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:19:29.963 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:49.603907+0000 37'3 37:50 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:17:44.581101+0000 0 0 2024-04-25T07:19:29.963 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:46.720794+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:09:14.685191+0000 0 0 2024-04-25T07:19:29.963 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:18:44.730861+0000 37'4 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:28:16.294094+0000 0 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:51.728026+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:04:51.414488+0000 0 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:56.743712+0000 37'3 37:58 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:53:44.268003+0000 1 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 0 0 0 3435 0 0 6 0 6 active+clean 2024-04-25T07:18:58.780868+0000 37'6 37:61 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:38:59.269300+0000 2 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:57.793345+0000 37'3 37:58 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:23:39.590419+0000 0 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:19:00.908033+0000 37'2 37:57 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:49:55.238267+0000 1 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:18:53.803820+0000 37'4 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:14:27.294452+0000 0 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-04-25T07:18:49.854281+0000 37'6 37:54 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:32:12.748825+0000 0 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:18:50.740506+0000 37'2 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:12:05.054924+0000 0 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-04-25T07:18:55.832851+0000 37'5 37:60 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:15:58.090184+0000 1 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:18:46.514859+0000 37'1 37:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:54:30.505712+0000 0 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:49.751722+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:43:50.607718+0000 0 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:18:46.815527+0000 37'2 37:50 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:52:56.067478+0000 0 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:18:53.670031+0000 37'2 37:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:25:54.867624+0000 0 0 2024-04-25T07:19:29.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:43.669542+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:32:10.336941+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:47.844714+0000 37'3 37:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:21:35.075870+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 0 0 0 1851 0 0 6 0 6 active+clean 2024-04-25T07:18:59.847125+0000 37'6 37:61 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:56:12.043910+0000 2 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:18:48.891102+0000 37'4 37:52 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:54:44.155493+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:48.591032+0000 37'3 37:50 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:29:27.576830+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 2 0 0 0 0 1234 0 0 4 0 4 active+clean 2024-04-25T07:18:47.552772+0000 37'4 37:51 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:59:09.784006+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:18:52.640478+0000 37'1 37:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:18:05.952824+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:45.842787+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:17:10.472671+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:50.875188+0000 37'3 37:51 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:33:34.607451+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:52.731903+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:14:36.657130+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:18:48.724703+0000 37'4 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:37:52.999573+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:18:51.601603+0000 37'1 37:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:47:57.800429+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:18:45.841071+0000 37'3 37:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:57:25.545147+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:18:47.700445+0000 37'2 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:15:42.736676+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:18:54.759791+0000 37'4 37:59 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:06:13.144443+0000 0 0 2024-04-25T07:19:29.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 6 0 0 0 0 3435 0 0 7 0 7 active+clean 2024-04-25T07:18:50.605832+0000 37'7 37:54 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:49:02.053366+0000 0 0 2024-04-25T07:19:29.966 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:19:29.966 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:19:29.966 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:19:29.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:19:29.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:19:29.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:19:29.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:19:29.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:19:29.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:19:30.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:19:30.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:19:30.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-04-25T07:19:30.645 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T07:19:30.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:19:30.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-04-25T07:19:31.279 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already down. 2024-04-25T07:19:31.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:19:31.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-04-25T07:19:31.922 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already down. 2024-04-25T07:19:31.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:19:31.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.1 2024-04-25T07:19:32.533 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already out. 2024-04-25T07:19:32.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:19:32.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-04-25T07:19:33.109 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already out. 2024-04-25T07:19:33.125 INFO:tasks.workunit.client.0.smithi102.stdout:bringing up non_divergent 1 2 2024-04-25T07:19:33.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 1 2' 2024-04-25T07:19:33.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:19:33.547 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:19:33.547 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:49.603907+0000 37'3 37:50 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:17:44.581101+0000 0 0 2024-04-25T07:19:33.547 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:46.720794+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:09:14.685191+0000 0 0 2024-04-25T07:19:33.547 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-04-25T07:18:44.730861+0000 37'4 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:28:16.294094+0000 0 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:51.728026+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:04:51.414488+0000 0 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:56.743712+0000 37'3 37:58 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:53:44.268003+0000 1 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 0 0 0 3435 0 0 6 0 6 stale+active+clean 2024-04-25T07:18:58.780868+0000 37'6 37:61 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:38:59.269300+0000 2 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:57.793345+0000 37'3 37:58 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:23:39.590419+0000 0 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-04-25T07:19:00.908033+0000 37'2 37:57 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:49:55.238267+0000 1 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-04-25T07:18:53.803820+0000 37'4 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:14:27.294452+0000 0 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 stale+active+clean 2024-04-25T07:18:49.854281+0000 37'6 37:54 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:32:12.748825+0000 0 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-04-25T07:18:50.740506+0000 37'2 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:12:05.054924+0000 0 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 stale+active+clean 2024-04-25T07:18:55.832851+0000 37'5 37:60 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:15:58.090184+0000 1 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 stale+active+clean 2024-04-25T07:18:46.514859+0000 37'1 37:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:54:30.505712+0000 0 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:49.751722+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:43:50.607718+0000 0 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-04-25T07:18:46.815527+0000 37'2 37:50 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:52:56.067478+0000 0 0 2024-04-25T07:19:33.548 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-04-25T07:18:53.670031+0000 37'2 37:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:25:54.867624+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:43.669542+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:32:10.336941+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:47.844714+0000 37'3 37:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:21:35.075870+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 0 0 0 1851 0 0 6 0 6 stale+active+clean 2024-04-25T07:18:59.847125+0000 37'6 37:61 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:56:12.043910+0000 2 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-04-25T07:18:48.891102+0000 37'4 37:52 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:54:44.155493+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:48.591032+0000 37'3 37:50 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:29:27.576830+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 2 0 0 0 0 1234 0 0 4 0 4 stale+active+clean 2024-04-25T07:18:47.552772+0000 37'4 37:51 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:59:09.784006+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 stale+active+clean 2024-04-25T07:18:52.640478+0000 37'1 37:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:18:05.952824+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:45.842787+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:17:10.472671+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:50.875188+0000 37'3 37:51 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:33:34.607451+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:52.731903+0000 37'3 37:54 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:14:36.657130+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-04-25T07:18:48.724703+0000 37'4 37:55 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:37:52.999573+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 stale+active+clean 2024-04-25T07:18:51.601603+0000 37'1 37:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:47:57.800429+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 stale+active+clean 2024-04-25T07:18:45.841071+0000 37'3 37:51 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:57:25.545147+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 stale+active+clean 2024-04-25T07:18:47.700445+0000 37'2 37:53 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:15:42.736676+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 stale+active+clean 2024-04-25T07:18:54.759791+0000 37'4 37:59 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:06:13.144443+0000 0 0 2024-04-25T07:19:33.549 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 6 0 0 0 0 3435 0 0 7 0 7 stale+active+clean 2024-04-25T07:18:50.605832+0000 37'7 37:54 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:49:02.053366+0000 0 0 2024-04-25T07:19:33.550 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:19:33.550 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:19:33.550 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:19:33.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:19:33.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-04-25T07:19:33.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:19:33.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:19:33.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:19:33.566 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:19:33.566 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:33.566 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:19:33.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:19:33.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:19:33.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:19:33.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:19:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:19:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:19:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:19:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:19:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:19:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:19:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:19:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:19:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:19:33.567 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:19:33.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:19:33.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:19:33.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:19:33.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:19:33.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:19:33.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:19:33.600 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:19:33.600+0000 7f3d96c727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:33.602 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:19:33.604+0000 7f3d96c727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:33.603 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:19:33.604+0000 7f3d96c727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:33.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:19:33.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:19:33.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:19:33.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:19:33.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:19:33.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:33.973 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:19:33.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:19:33.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:33.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:34.155 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:19:34.156+0000 7f3d96c727c0 -1 Falling back to public interface 2024-04-25T07:19:34.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:35.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:35.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:35.404 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:19:35.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:19:35.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:35.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:35.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:36.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:36.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:36.837 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:19:36.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:19:36.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:36.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:37.161 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:19:37.164+0000 7f3d96c727c0 -1 osd.1 37 log_to_monitors true 2024-04-25T07:19:37.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:38.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:38.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:38.277 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:19:38.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:19:38.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:38.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:19:38.721 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up out weight 0 up_from 44 up_thru 35 down_at 39 last_clean_interval [10,37) [v2:127.0.0.1:6802/550444832,v1:127.0.0.1:6803/550444832] [v2:127.0.0.1:6804/550444832,v1:127.0.0.1:6805/550444832] exists,up df369e44-fe63-4f08-84e4-07f167676b56 2024-04-25T07:19:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:19:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:19:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:19:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:19:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-04-25T07:19:38.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:19:38.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:19:38.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:19:38.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:19:38.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:19:38.727 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:19:38.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:19:38.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:19:38.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:19:38.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:19:38.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:19:38.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:19:38.770 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:19:38.772+0000 7f5bed41d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:38.772 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:19:38.772+0000 7f5bed41d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:19:38.776+0000 7f5bed41d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:19:39.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:19:39.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:19:39.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:19:39.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:19:39.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:19:39.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:39.155 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:19:39.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:19:39.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:39.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:39.319 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:19:39.320+0000 7f5bed41d7c0 -1 Falling back to public interface 2024-04-25T07:19:39.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:40.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:40.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:40.589 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:19:40.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:19:40.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:40.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:41.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:42.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:42.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:42.010 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:19:42.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:19:42.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:42.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:42.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:42.714 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:19:42.716+0000 7f5bed41d7c0 -1 osd.2 37 log_to_monitors true 2024-04-25T07:19:43.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:43.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:43.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:19:43.479 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:19:43.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:43.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:43.958 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up out weight 0 up_from 46 up_thru 35 down_at 40 last_clean_interval [15,37) [v2:127.0.0.1:6810/2940722690,v1:127.0.0.1:6811/2940722690] [v2:127.0.0.1:6812/2940722690,v1:127.0.0.1:6813/2940722690] exists,up dcba27a8-55a9-40c9-9cc7-d09a609befd5 2024-04-25T07:19:43.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:19:43.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:19:43.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:19:43.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:19:43.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.1 2024-04-25T07:19:44.469 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already in. 2024-04-25T07:19:44.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:19:44.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-04-25T07:19:45.546 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already in. 2024-04-25T07:19:45.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:19:45.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-04-25T07:19:45.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:19:45.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:19:45.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:19:45.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:19:45.571 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:19:45.571 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:19:45.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:19:45.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:19:45.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:19:45.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:19:45.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:19:45.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:19:45.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:19:45.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:19:45.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:19:46.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:19:46.279 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:19:46.279 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:19:46.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:19:46.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:46.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:19:46.444 INFO:tasks.workunit.client.0.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T07:19:46.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:19:46.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:19:46.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:19:46.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:46.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:19:46.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561029 2024-04-25T07:19:46.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561029 2024-04-25T07:19:46.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-188978561029' 2024-04-25T07:19:46.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:46.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:19:46.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=197568495619 2024-04-25T07:19:46.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 197568495619 2024-04-25T07:19:46.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-188978561029 2-197568495619' 2024-04-25T07:19:46.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:46.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-188978561029 2024-04-25T07:19:46.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:46.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:19:46.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-188978561029 2024-04-25T07:19:46.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:46.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561029 2024-04-25T07:19:46.843 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 188978561029 2024-04-25T07:19:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 188978561029' 2024-04-25T07:19:46.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:19:47.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561027 -lt 188978561029 2024-04-25T07:19:47.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:19:48.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:19:48.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:19:48.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561029 -lt 188978561029 2024-04-25T07:19:48.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:48.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-197568495619 2024-04-25T07:19:48.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:48.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:19:48.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-197568495619 2024-04-25T07:19:48.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:48.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=197568495619 2024-04-25T07:19:48.688 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 197568495619 2024-04-25T07:19:48.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 197568495619' 2024-04-25T07:19:48.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:19:49.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 197568495619 -lt 197568495619 2024-04-25T07:19:49.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:19:49.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:49.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:49.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:19:49.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:19:49.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:19:49.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:19:49.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:19:49.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:19:49.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:19:49.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:19:50.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:19:50.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:19:50.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:50.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:50.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:19:50.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:19:50.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:19:50.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:19:50.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:19:50.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:19:50.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:19:50.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:19:50.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:19:50.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:19:50.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:19:50.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:19:50.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:19:50.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:19:51.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:19:51.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:19:51.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:51.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:51.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:19:51.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:19:51.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:19:51.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:19:51.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:19:51.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:19:51.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:19:51.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:19:51.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:19:52.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:19:52.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:19:52.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:19:52.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:19:52.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:19:52.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:19:52.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:19:52.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:19:52.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:19:52.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:19:52.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:19:52.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:19:52.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:19:52.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:19:52.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:19:52.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:52.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:53.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:19:53.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:19:53.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:19:53.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:19:53.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:19:53.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:19:53.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:19:53.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:19:53.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:19:54.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:19:54.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:19:54.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:19:54.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:19:54.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:19:54.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:19:54.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:19:54.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:19:54.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:19:54.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:19:54.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:19:54.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:19:54.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:19:54.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:19:54.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:19:54.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:54.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:19:55.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:19:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:19:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:19:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:19:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:19:55.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:19:55.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:19:55.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:19:56.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:19:56.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:19:56.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:19:56.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:19:56.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:19:56.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:19:56.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:19:56.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:19:56.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:19:56.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:19:56.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:19:56.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:19:56.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:19:57.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:19:57.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:19:57.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:57.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:57.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:19:57.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:19:57.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:19:57.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:19:57.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:19:57.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:19:57.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:19:57.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:19:57.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:19:58.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:19:58.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:19:58.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:19:58.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:19:58.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:20:00.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:00.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:00.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:00.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:00.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:00.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:00.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:00.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:00.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:00.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:00.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:00.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:01.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:01.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:01.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:01.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:01.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:01.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:01.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:01.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:01.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:01.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:01.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:01.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:20:01.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:01.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:20:04.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:04.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:04.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:04.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:04.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:04.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:04.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:04.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:05.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:05.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:05.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:05.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:05.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:05.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:05.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:05.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:05.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:05.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:05.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:05.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:05.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:06.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:06.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:06.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:20:06.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:06.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:20:12.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:12.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:12.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:12.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:12.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:12.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:12.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:12.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:13.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:13.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:13.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:13.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:13.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:13.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:14.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:14.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:14.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:20:14.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:14.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:20:21.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:21.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:21.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:21.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:21.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:21.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:21.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:21.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:22.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:22.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:22.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:22.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:22.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:22.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:22.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:22.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:22.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:22.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:22.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:22.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:22.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:23.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:23.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:23.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:20:23.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_fingerprint": "11e43376-f6a2-4d16-b88c-9b2efe1f7a73", 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout: "timestamp": "2024-04-25T07:20:23.821767+0000", 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout: "tag": "", 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout: "health": { 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout: "status": "HEALTH_WARN", 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout: "checks": { 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:20:23.819 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 17 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [], 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: "OSD_DOWN": { 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 osds down", 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "osd.0 (root=default,host=smithi102) is down" 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.820 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: "PG_DEGRADED": { 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 32 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,1]" 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.821 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,1]" 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.822 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.a is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.b is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.c is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.823 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,1]" 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.e is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,1]" 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [2,1]" 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.824 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [2,1]" 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.825 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,1]" 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.826 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [1,2]" 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,1]" 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:20:23.827 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: "mutes": [] 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_first_committed": 1, 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_last_committed": 1, 2024-04-25T07:20:23.828 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap": { 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:16:33.178368Z", 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:16:33.178368Z", 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release": 19, 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release_name": "squid", 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "election_strategy": 1, 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "disallowed_leaders: ": "", 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": false, 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "tiebreaker_mon": "", 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_ranks: ": "", 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "features": { 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "persistent": [ 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "kraken", 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "luminous", 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "mimic", 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap-prune", 2024-04-25T07:20:23.829 INFO:tasks.workunit.client.0.smithi102.stdout: "nautilus", 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "octopus", 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "pacific", 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "elector-pinging", 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "quincy", 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "reef", 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "squid" 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "optional": [] 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "mons": [ 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "rank": 0, 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "a", 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.830 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 0 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 0, 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 0, 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_location": "{}" 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.831 INFO:tasks.workunit.client.0.smithi102.stdout: "quorum": [ 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: 0 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap": { 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 52, 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:16:34.168669+0000", 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:19:46.353111+0000", 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: "last_up_change": "2024-04-25T07:19:43.118190+0000", 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: "last_in_change": "2024-04-25T07:19:45.073869+0000", 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_num": 5800384, 2024-04-25T07:20:23.832 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_set": [ 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "nodown", 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "noin", 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "noout", 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "pglog_hardlimit", 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snapdirs", 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "recovery_deletes", 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "sortbitwise" 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_version": 9, 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_snapshot": "", 2024-04-25T07:20:23.833 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_max": 1, 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "max_osd": 3, 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "min_compat_client": "jewel", 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "require_osd_release": "squid", 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "allow_crimson": false, 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "pools": [ 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_name": "test", 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "create_time": "2024-04-25T07:18:29.374110+0000", 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": 1, 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_names": "hashpspool", 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 3, 2024-04-25T07:20:23.834 INFO:tasks.workunit.client.0.smithi102.stdout: "min_size": 2, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_rule": 0, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "object_hash": 2, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_autoscale_mode": "on", 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num": 32, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num": 32, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num_target": 32, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_target": 32, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_pending": 32, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pg_merge_meta": { 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "source_pgid": "0.0", 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "ready_epoch": 0, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_started": 0, 2024-04-25T07:20:23.835 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": 0, 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "source_version": "0'0", 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "target_version": "0'0" 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "last_change": "35", 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend": "0", 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_preluminous": "33", 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "auid": 0, 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_seq": 0, 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_epoch": 0, 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_snaps": [], 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps": "[]", 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_bytes": 0, 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_objects": 0, 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "tiers": [], 2024-04-25T07:20:23.836 INFO:tasks.workunit.client.0.smithi102.stdout: "tier_of": -1, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "read_tier": -1, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "write_tier": -1, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_mode": "none", 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_bytes": 0, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_objects": 0, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_flush_age": 0, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_evict_age": 0, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profile": "", 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_params": { 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "none" 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_period": 0, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_count": 0, 2024-04-25T07:20:23.837 INFO:tasks.workunit.client.0.smithi102.stdout: "use_gmt_hitset": true, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "grade_table": [], 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "stripe_width": 0, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_num_objects": 0, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "fast_read": false, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "options": {}, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "application_metadata": {}, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "read_balance": { 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "score_type": "Fair distribution", 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "score_acting": 2.2400000095367432, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "score_stable": 2.2400000095367432, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:20:23.838 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_acting": 1.5, 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_stable": 1.5, 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity": 1, 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "osds": [ 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "3d1dc328-65a0-4947-a470-7ad55360f6cb", 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 0, 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:20:23.839 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 0, 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 0, 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 5, 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 35, 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 38, 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:20:23.840 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.841 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:20:23.842 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6803/640311997", 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6805/640311997", 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6809/640311997", 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6807/640311997", 2024-04-25T07:20:23.843 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "exists" 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "df369e44-fe63-4f08-84e4-07f167676b56", 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 10, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 37, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 44, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 49, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 39, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:20:23.844 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.845 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:20:23.846 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.847 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6803/550444832", 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6805/550444832", 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6809/550444832", 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6807/550444832", 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.848 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "dcba27a8-55a9-40c9-9cc7-d09a609befd5", 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 15, 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 37, 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 46, 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 51, 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 40, 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:20:23.849 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.850 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:20:23.851 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.852 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/2940722690", 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/2940722690", 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2940722690", 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2940722690", 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_xinfo": [ 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:19:30.445148+0000", 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:20:23.853 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:17:09.775357+0000", 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:19:31.081554+0000", 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:17:46.953846+0000", 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:19:31.717282+0000", 2024-04-25T07:20:23.854 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:18:24.793145+0000", 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap": [], 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_items": [], 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_primaries": [], 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_temp": [], 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_temp": [], 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "blocklist": {}, 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "range_blocklist": {}, 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profiles": { 2024-04-25T07:20:23.855 INFO:tasks.workunit.client.0.smithi102.stdout: "default": { 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "k": "2", 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "m": "2", 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "plugin": "jerasure", 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "technique": "reed_sol_van" 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps_queue": [], 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "new_removed_snaps": [], 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "new_purged_snaps": [], 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_node_flags": {}, 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "device_class_flags": {}, 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": { 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_enabled": false, 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_bucket_count": 0, 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:20:23.856 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_bucket": 0 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_metadata": [ 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6804/640311997,v1:127.0.0.1:6805/640311997]", 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:20:23.857 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:16:41.127782Z", 2024-04-25T07:20:23.858 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6802/640311997,v1:127.0.0.1:6803/640311997]", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/640311997,v1:127.0.0.1:6809/640311997]", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/640311997,v1:127.0.0.1:6807/640311997]", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:20:23.859 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:20:23.860 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6804/550444832,v1:127.0.0.1:6805/550444832]", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:20:23.861 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:17:16.875500Z", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:20:23.862 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6802/550444832,v1:127.0.0.1:6803/550444832]", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/550444832,v1:127.0.0.1:6809/550444832]", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/550444832,v1:127.0.0.1:6807/550444832]", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/2940722690,v1:127.0.0.1:6813/2940722690]", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:20:23.864 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:17:55.097579Z", 2024-04-25T07:20:23.865 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/2940722690,v1:127.0.0.1:6811/2940722690]", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2940722690,v1:127.0.0.1:6817/2940722690]", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2940722690,v1:127.0.0.1:6815/2940722690]", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:20:23.866 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_clean_epochs": { 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": { 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "per_pool": [ 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.867 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: "floor": 34 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_epochs": [ 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 44 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 46 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_first_committed": 1, 2024-04-25T07:20:23.868 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_last_committed": 52, 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: "crushmap": { 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": [ 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.0", 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.1", 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.2", 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:20:23.869 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: "types": [ 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 0, 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd" 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "host" 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 2, 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "chassis" 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 3, 2024-04-25T07:20:23.870 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "rack" 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 4, 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "row" 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 5, 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pdu" 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 6, 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pod" 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 7, 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "room" 2024-04-25T07:20:23.871 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 8, 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "datacenter" 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 9, 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "zone" 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 10, 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "region" 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:20:23.872 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "root" 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "buckets": [ 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -1, 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default", 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:20:23.873 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -2, 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default~hdd", 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:20:23.874 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102", 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:20:23.875 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102~hdd", 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:20:23.876 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.877 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: "rules": [ 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_id": 0, 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_name": "replicated_rule", 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: "steps": [ 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.878 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "take", 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: "item": -1, 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: "item_name": "default" 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "choose_firstn", 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 0, 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "osd" 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "emit" 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: "tunables": { 2024-04-25T07:20:23.879 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_tries": 0, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_total_tries": 50, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_stable": 1, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_descents": 100, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_collision_tries": 100, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "straw_calc_version": 1, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "profile": "jewel", 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_tunables": 1, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "legacy_tunables": 0, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "minimum_required_version": "jewel", 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables": 1, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables2": 1, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v2_rules": 0, 2024-04-25T07:20:23.880 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables3": 1, 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v3_rules": 0, 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v4_buckets": 1, 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables5": 1, 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v5_rules": 0, 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "has_msr_rules": 0 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_args": {} 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "fsmap": { 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "default_fscid": -1, 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": { 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": {}, 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "ro_compat": {}, 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "incompat": { 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_1": "base v0.20", 2024-04-25T07:20:23.881 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_8": "no anchor table", 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_9": "file layout v2", 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_11": "minor log segments", 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_flags": { 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "enable_multiple": true, 2024-04-25T07:20:23.882 INFO:tasks.workunit.client.0.smithi102.stdout: "ever_enabled_multiple": true 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "standbys": [], 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "filesystems": [] 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "auth": { 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 5, 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "num_secrets": 3 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 32, 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_active": 32, 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_unknown": 0, 2024-04-25T07:20:23.883 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osd": 3, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_sum": { 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 61700, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 100, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 300, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 100, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 100, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 93, 2024-04-25T07:20:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 93, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:20:23.885 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:20:23.886 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 0, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 0, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 100, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 100, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 64, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 64, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 0 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum": { 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 96, 2024-04-25T07:20:23.887 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83856, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2344, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314488944, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036678656, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2400256, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 550377, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:20:23.888 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10392, 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416936 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:20:23.889 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum_by_class": { 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "hdd": { 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 96, 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83856, 2024-04-25T07:20:23.890 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2344, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314488944, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036678656, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2400256, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 550377, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10392, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416936 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:20:23.891 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:20:23.892 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_stats": [ 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 32, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 61700, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 100, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 300, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 100, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:20:23.893 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 100, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 93, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 93, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:20:23.894 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:20:23.895 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 1220608, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 184566, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:20:23.896 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 100, 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 100, 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 64, 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 64, 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 3 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_stats": [ 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 21474836521 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 188978561037 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.897 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 197568495626 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_state": [ 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "active+undersized+degraded", 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 32 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_osd": [ 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.898 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 8, 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 32, 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 0, 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 24, 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 32, 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:20:23.899 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": [ 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": {} 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "servicemap": { 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 5, 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:19:27.573118+0000", 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "services": {} 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_last_committed": 104, 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_first_committed": 1, 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_last_committed": 443, 2024-04-25T07:20:23.900 INFO:tasks.workunit.client.0.smithi102.stdout: "paxos": { 2024-04-25T07:20:23.901 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:20:23.901 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 602, 2024-04-25T07:20:23.901 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pn": 0, 2024-04-25T07:20:23.901 INFO:tasks.workunit.client.0.smithi102.stdout: "accepted_pn": 0 2024-04-25T07:20:23.901 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:20:23.901 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:20:23.901 INFO:tasks.workunit.client.0.smithi102.stderr:report 2592704069 2024-04-25T07:20:23.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:20:23.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-04-25T07:20:23.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-04-25T07:20:23.902 INFO:tasks.workunit.client.0.smithi102.stdout:writing non-divergent object existing_4 2024-04-25T07:20:23.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-04-25T07:20:23.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:20:24.255 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:20:24.255 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.644633+0000 37'3 52:57 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:48:13.811063+0000 0 0 2024-04-25T07:20:24.255 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.891189+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:07:11.426953+0000 0 0 2024-04-25T07:20:24.255 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.892628+0000 37'4 52:68 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:52:07.878867+0000 0 0 2024-04-25T07:20:24.255 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.897204+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:51:30.387558+0000 0 0 2024-04-25T07:20:24.255 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892039+0000 37'3 52:71 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:01:00.093189+0000 1 0 2024-04-25T07:20:24.255 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.895656+0000 37'5 52:73 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:41.312970+0000 2 0 2024-04-25T07:20:24.255 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892884+0000 37'3 52:71 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:49:26.167660+0000 0 0 2024-04-25T07:20:24.255 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.894574+0000 37'2 52:70 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:22:07.893974+0000 1 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.895776+0000 37'4 52:68 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:33.686310+0000 0 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:19:46.758210+0000 37'6 52:61 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:15:06.486943+0000 0 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.893616+0000 37'2 52:66 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:51:34.511222+0000 0 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891839+0000 37'5 52:73 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:06.657091+0000 1 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.891804+0000 37'1 52:61 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:08:31.240485+0000 0 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.893188+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:06:33.182195+0000 0 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:46.757106+0000 37'2 52:57 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:49:50.049657+0000 0 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.894954+0000 37'2 52:62 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:14.888489+0000 0 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.850172+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:53:23.706783+0000 0 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755995+0000 37'3 52:58 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:27:37.647191+0000 0 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.891582+0000 37'4 52:72 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:16:25.175530+0000 2 0 2024-04-25T07:20:24.256 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:46.757179+0000 37'4 52:59 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:53:26.802539+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.756164+0000 37'3 52:57 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:16.869210+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892306+0000 37'3 52:63 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:38:24.652869+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.895994+0000 37'1 52:61 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:34:12.123814+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894712+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:23:29.794915+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.756147+0000 37'3 52:58 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:14:48.298332+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894289+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:58:16.590827+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.891035+0000 37'4 52:68 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:06:11.586560+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.893909+0000 37'1 52:61 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:27:15.069777+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755975+0000 37'3 52:58 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:31.965147+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.892863+0000 37'2 52:66 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:47:36.332515+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.896539+0000 37'4 52:72 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:22:04.797910+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.892753+0000 37'5 52:65 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:11:10.391476+0000 0 0 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:20:24.257 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:20:24.258 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:20:24.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:20:24.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:20:24.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-04-25T07:20:24.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:20:24.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:20:24.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:20:24.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:20:24.405 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:20:24.405 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:20:24.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:20:24.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:20:24.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:20:24.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:20:24.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:20:24.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:20:24.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:20:24.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:20:24.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:20:24.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:20:24.897 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:20:24.897 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:20:24.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:20:24.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:24.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:20:25.035 INFO:tasks.workunit.client.0.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T07:20:25.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:20:25.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:20:25.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:20:25.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:25.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:20:25.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=188978561039 2024-04-25T07:20:25.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 188978561039 2024-04-25T07:20:25.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-188978561039' 2024-04-25T07:20:25.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:25.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:20:25.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=197568495629 2024-04-25T07:20:25.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 197568495629 2024-04-25T07:20:25.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-188978561039 2-197568495629' 2024-04-25T07:20:25.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:20:25.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-188978561039 2024-04-25T07:20:25.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:20:25.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:20:25.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-188978561039 2024-04-25T07:20:25.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:20:25.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=188978561039 2024-04-25T07:20:25.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 188978561039' 2024-04-25T07:20:25.348 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 188978561039 2024-04-25T07:20:25.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:20:25.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 188978561039 -lt 188978561039 2024-04-25T07:20:25.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:20:25.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-197568495629 2024-04-25T07:20:25.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:20:25.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:20:25.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-197568495629 2024-04-25T07:20:25.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:20:25.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=197568495629 2024-04-25T07:20:25.763 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 197568495629 2024-04-25T07:20:25.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 197568495629' 2024-04-25T07:20:25.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:20:26.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 197568495629 -lt 197568495629 2024-04-25T07:20:26.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:20:26.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:26.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:20:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:26.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:26.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:26.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:26.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:26.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:26.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:27.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:27.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:27.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:27.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:27.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:27.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:20:27.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:20:27.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:20:27.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:27.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:20:27.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:27.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:27.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:27.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:27.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:27.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:27.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:27.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:28.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:28.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:28.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:28.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:28.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:28.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:28.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:28.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:28.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:28.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:28.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:28.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:28.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:29.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:29.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:29.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:20:29.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:29.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:20:29.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:29.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:29.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:29.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:29.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:29.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:29.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:29.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:30.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:30.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:30.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:30.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:30.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:30.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:30.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:30.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:30.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:30.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:30.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:30.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:30.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:31.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:31.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:31.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:20:31.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:31.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:20:31.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:31.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:31.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:31.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:31.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:31.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:31.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:31.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:31.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:31.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:31.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:31.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:32.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:32.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:33.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:33.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:33.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:20:33.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:33.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:20:33.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:33.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:33.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:33.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:33.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:33.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:33.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:33.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:34.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:34.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:34.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:34.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:34.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:34.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:34.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:34.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:34.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:34.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:34.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:34.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:34.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:20:37.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:37.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:37.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:37.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:37.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:37.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:37.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:37.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:37.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:37.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:37.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:37.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:38.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:38.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:38.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:38.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:38.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:38.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:38.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:38.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:38.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:38.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:38.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:38.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:20:38.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:38.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:20:41.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:41.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:41.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:41.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:41.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:41.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:41.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:41.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:42.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:42.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:42.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:42.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:42.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:42.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:42.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:42.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:42.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:42.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:42.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:42.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:42.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:20:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:20:49.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:49.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:49.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:49.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:49.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:49.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:49.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:49.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:50.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:50.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:50.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:50.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:50.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:50.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:50.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:50.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:51.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:51.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:51.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:20:51.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:51.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:20:58.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:58.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:58.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:58.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:58.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:58.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:58.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:58.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:59.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:59.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:59.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:59.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:59.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:59.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:59.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:59.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:59.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:59.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:59.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:59.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:59.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:21:00.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:21:00.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:21:00.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:21:00.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:21:00.831 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:21:00.832 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_fingerprint": "11e43376-f6a2-4d16-b88c-9b2efe1f7a73", 2024-04-25T07:21:00.832 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:21:00.832 INFO:tasks.workunit.client.0.smithi102.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:21:00.832 INFO:tasks.workunit.client.0.smithi102.stdout: "timestamp": "2024-04-25T07:21:00.834532+0000", 2024-04-25T07:21:00.832 INFO:tasks.workunit.client.0.smithi102.stdout: "tag": "", 2024-04-25T07:21:00.832 INFO:tasks.workunit.client.0.smithi102.stdout: "health": { 2024-04-25T07:21:00.833 INFO:tasks.workunit.client.0.smithi102.stdout: "status": "HEALTH_WARN", 2024-04-25T07:21:00.833 INFO:tasks.workunit.client.0.smithi102.stdout: "checks": { 2024-04-25T07:21:00.833 INFO:tasks.workunit.client.0.smithi102.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:21:00.833 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:21:00.833 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:21:00.833 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:21:00.833 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 17 2024-04-25T07:21:00.833 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.834 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [], 2024-04-25T07:21:00.834 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:21:00.834 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.834 INFO:tasks.workunit.client.0.smithi102.stdout: "OSD_DOWN": { 2024-04-25T07:21:00.834 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:21:00.834 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:21:00.834 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 osds down", 2024-04-25T07:21:00.835 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:21:00.835 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.836 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:21:00.836 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.836 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "osd.0 (root=default,host=smithi102) is down" 2024-04-25T07:21:00.836 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.836 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.836 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:21:00.837 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.837 INFO:tasks.workunit.client.0.smithi102.stdout: "PG_DEGRADED": { 2024-04-25T07:21:00.837 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:21:00.837 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:21:00.837 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-04-25T07:21:00.837 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 64 2024-04-25T07:21:00.837 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.838 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:21:00.838 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.838 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.0 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.838 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.838 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.838 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-04-25T07:21:00.838 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.838 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.2 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.3 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.4 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.5 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.6 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.839 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.7 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.8 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.9 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.a is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.b is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.840 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.c is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.d is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.e is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.f is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.10 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-04-25T07:21:00.841 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.11 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.12 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.13 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.14 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.15 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.842 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.16 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.17 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.18 is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.19 is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1a is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.843 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1b is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1c is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1d is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1e is stuck undersized for 73s, current state active+undersized+degraded, last acting [1,2]" 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.1f is stuck undersized for 73s, current state active+undersized+degraded, last acting [2,1]" 2024-04-25T07:21:00.844 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.845 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: "mutes": [] 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_first_committed": 1, 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_last_committed": 1, 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap": { 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:16:33.178368Z", 2024-04-25T07:21:00.846 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:16:33.178368Z", 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release": 19, 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release_name": "squid", 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "election_strategy": 1, 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "disallowed_leaders: ": "", 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": false, 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "tiebreaker_mon": "", 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_ranks: ": "", 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "features": { 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "persistent": [ 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "kraken", 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "luminous", 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "mimic", 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap-prune", 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "nautilus", 2024-04-25T07:21:00.847 INFO:tasks.workunit.client.0.smithi102.stdout: "octopus", 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "pacific", 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "elector-pinging", 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "quincy", 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "reef", 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "squid" 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "optional": [] 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "mons": [ 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "rank": 0, 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "a", 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.848 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 0 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 0, 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 0, 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_location": "{}" 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: "quorum": [ 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: 0 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.849 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap": { 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 52, 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:16:34.168669+0000", 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:19:46.353111+0000", 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "last_up_change": "2024-04-25T07:19:43.118190+0000", 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "last_in_change": "2024-04-25T07:19:45.073869+0000", 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_num": 5800384, 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_set": [ 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "nodown", 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "noin", 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "noout", 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "pglog_hardlimit", 2024-04-25T07:21:00.850 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snapdirs", 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "recovery_deletes", 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "sortbitwise" 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_version": 9, 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_snapshot": "", 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_max": 1, 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "max_osd": 3, 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "min_compat_client": "jewel", 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "require_osd_release": "squid", 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "allow_crimson": false, 2024-04-25T07:21:00.851 INFO:tasks.workunit.client.0.smithi102.stdout: "pools": [ 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_name": "test", 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "create_time": "2024-04-25T07:18:29.374110+0000", 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": 1, 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_names": "hashpspool", 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 3, 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "min_size": 2, 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_rule": 0, 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "object_hash": 2, 2024-04-25T07:21:00.852 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_autoscale_mode": "on", 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num": 32, 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num": 32, 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num_target": 32, 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_target": 32, 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_pending": 32, 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pg_merge_meta": { 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "source_pgid": "0.0", 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "ready_epoch": 0, 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_started": 0, 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": 0, 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "source_version": "0'0", 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "target_version": "0'0" 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "last_change": "35", 2024-04-25T07:21:00.853 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend": "0", 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_preluminous": "33", 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "auid": 0, 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_seq": 0, 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_epoch": 0, 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_snaps": [], 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps": "[]", 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_bytes": 0, 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_objects": 0, 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "tiers": [], 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "tier_of": -1, 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "read_tier": -1, 2024-04-25T07:21:00.854 INFO:tasks.workunit.client.0.smithi102.stdout: "write_tier": -1, 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_mode": "none", 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_bytes": 0, 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_objects": 0, 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_flush_age": 0, 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_evict_age": 0, 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profile": "", 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_params": { 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "none" 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_period": 0, 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_count": 0, 2024-04-25T07:21:00.855 INFO:tasks.workunit.client.0.smithi102.stdout: "use_gmt_hitset": true, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "grade_table": [], 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "stripe_width": 0, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_num_objects": 0, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "fast_read": false, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "options": {}, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "application_metadata": {}, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "read_balance": { 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "score_type": "Fair distribution", 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "score_acting": 2.2400000095367432, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "score_stable": 2.2400000095367432, 2024-04-25T07:21:00.856 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_acting": 1.5, 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_stable": 1.5, 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity": 1, 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: "osds": [ 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "3d1dc328-65a0-4947-a470-7ad55360f6cb", 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 0, 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:21:00.857 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 0, 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 0, 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 5, 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 35, 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 38, 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.858 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.859 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.860 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.861 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 640311997 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6803/640311997", 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6805/640311997", 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6809/640311997", 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6807/640311997", 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: "exists" 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.862 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "df369e44-fe63-4f08-84e4-07f167676b56", 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 10, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 37, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 44, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 49, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 39, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:21:00.863 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.864 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.865 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.866 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 550444832 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6803/550444832", 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6805/550444832", 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6809/550444832", 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6807/550444832", 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:21:00.867 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "dcba27a8-55a9-40c9-9cc7-d09a609befd5", 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 15, 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 37, 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 46, 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 51, 2024-04-25T07:21:00.868 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 40, 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.869 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.870 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:21:00.871 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2940722690 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/2940722690", 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/2940722690", 2024-04-25T07:21:00.872 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2940722690", 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2940722690", 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_xinfo": [ 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:19:30.445148+0000", 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:21:00.873 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:17:09.775357+0000", 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:19:31.081554+0000", 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:17:46.953846+0000", 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.874 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:19:31.717282+0000", 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:18:24.793145+0000", 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap": [], 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_items": [], 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_primaries": [], 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_temp": [], 2024-04-25T07:21:00.875 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_temp": [], 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "blocklist": {}, 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "range_blocklist": {}, 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profiles": { 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "default": { 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "k": "2", 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "m": "2", 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "plugin": "jerasure", 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "technique": "reed_sol_van" 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps_queue": [], 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "new_removed_snaps": [], 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "new_purged_snaps": [], 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_node_flags": {}, 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "device_class_flags": {}, 2024-04-25T07:21:00.876 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": { 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_enabled": false, 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_bucket_count": 0, 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_bucket": 0 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_metadata": [ 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6804/640311997,v1:127.0.0.1:6805/640311997]", 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:21:00.877 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:21:00.878 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:16:41.127782Z", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6802/640311997,v1:127.0.0.1:6803/640311997]", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:21:00.879 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/640311997,v1:127.0.0.1:6809/640311997]", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/640311997,v1:127.0.0.1:6807/640311997]", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:21:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6804/550444832,v1:127.0.0.1:6805/550444832]", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:21:00.881 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:17:16.875500Z", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:21:00.882 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6802/550444832,v1:127.0.0.1:6803/550444832]", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/550444832,v1:127.0.0.1:6809/550444832]", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/550444832,v1:127.0.0.1:6807/550444832]", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:21:00.883 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/2940722690,v1:127.0.0.1:6813/2940722690]", 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:21:00.884 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:00.885 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:17:55.097579Z", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/2940722690,v1:127.0.0.1:6811/2940722690]", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2940722690,v1:127.0.0.1:6817/2940722690]", 2024-04-25T07:21:00.886 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2940722690,v1:127.0.0.1:6815/2940722690]", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.887 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_clean_epochs": { 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": { 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "per_pool": [ 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "floor": 34 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_epochs": [ 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 44 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 46 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_first_committed": 1, 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_last_committed": 52, 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "crushmap": { 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": [ 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.0", 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.1", 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.2", 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "types": [ 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 0, 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd" 2024-04-25T07:21:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "host" 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 2, 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "chassis" 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 3, 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "rack" 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 4, 2024-04-25T07:21:00.891 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "row" 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 5, 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pdu" 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 6, 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pod" 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 7, 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "room" 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.892 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 8, 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "datacenter" 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 9, 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "zone" 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 10, 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "region" 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "root" 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.893 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.894 INFO:tasks.workunit.client.0.smithi102.stdout: "buckets": [ 2024-04-25T07:21:00.894 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.894 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -1, 2024-04-25T07:21:00.894 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default", 2024-04-25T07:21:00.894 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:21:00.894 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:21:00.894 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:21:00.894 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:21:00.894 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:21:00.894 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -2, 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default~hdd", 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:21:00.895 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102", 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:21:00.896 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:21:00.897 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102~hdd", 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.898 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.899 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: "rules": [ 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_id": 0, 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_name": "replicated_rule", 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: "steps": [ 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "take", 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: "item": -1, 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: "item_name": "default" 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "choose_firstn", 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 0, 2024-04-25T07:21:00.900 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "osd" 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "emit" 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: "tunables": { 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_tries": 0, 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_total_tries": 50, 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_stable": 1, 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_descents": 100, 2024-04-25T07:21:00.901 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_collision_tries": 100, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "straw_calc_version": 1, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "profile": "jewel", 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_tunables": 1, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "legacy_tunables": 0, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "minimum_required_version": "jewel", 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables": 1, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables2": 1, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v2_rules": 0, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables3": 1, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v3_rules": 0, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v4_buckets": 1, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables5": 1, 2024-04-25T07:21:00.902 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v5_rules": 0, 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "has_msr_rules": 0 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_args": {} 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "fsmap": { 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "default_fscid": -1, 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": { 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": {}, 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "ro_compat": {}, 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "incompat": { 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_1": "base v0.20", 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:21:00.903 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_8": "no anchor table", 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_9": "file layout v2", 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_11": "minor log segments", 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_flags": { 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "enable_multiple": true, 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "ever_enabled_multiple": true 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "standbys": [], 2024-04-25T07:21:00.904 INFO:tasks.workunit.client.0.smithi102.stdout: "filesystems": [] 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "auth": { 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 5, 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "num_secrets": 3 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 32, 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_active": 32, 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_unknown": 0, 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osd": 3, 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_sum": { 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 62050, 2024-04-25T07:21:00.905 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 100, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 300, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 100, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 100, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 94, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 94, 2024-04-25T07:21:00.906 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:21:00.907 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 0, 2024-04-25T07:21:00.908 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 0, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 101, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 101, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 64, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 64, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 0 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum": { 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:21:00.909 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 96, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83856, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2344, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314488944, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036678656, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:21:00.910 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2400256, 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 551077, 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10392, 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416936 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:21:00.911 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum_by_class": { 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "hdd": { 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 96, 2024-04-25T07:21:00.912 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83856, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2344, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314488944, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036678656, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2400256, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 551077, 2024-04-25T07:21:00.913 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10392, 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416936 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:21:00.914 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_stats": [ 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 32, 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 62050, 2024-04-25T07:21:00.915 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 100, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 300, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 100, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 100, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 94, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 94, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:21:00.916 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:21:00.917 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 1220608, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 185266, 2024-04-25T07:21:00.918 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 101, 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 101, 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 64, 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 64, 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 3 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_stats": [ 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.919 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:21:00.920 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 21474836521 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 188978561046 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 197568495636 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_state": [ 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "active+undersized+degraded", 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 32 2024-04-25T07:21:00.921 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_osd": [ 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 8, 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 32, 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 0, 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:21:00.922 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 24, 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 32, 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": [ 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": {} 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: "servicemap": { 2024-04-25T07:21:00.923 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 7, 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:20:45.601619+0000", 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: "services": {} 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_last_committed": 122, 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_first_committed": 1, 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_last_committed": 510, 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: "paxos": { 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 687, 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pn": 0, 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: "accepted_pn": 0 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:21:00.924 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:21:00.925 INFO:tasks.workunit.client.0.smithi102.stderr:report 840143973 2024-04-25T07:21:00.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:21:00.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-04-25T07:21:00.925 INFO:tasks.workunit.client.0.smithi102.stdout:delay recovery 2024-04-25T07:21:00.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:01.277 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:01.277 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.644633+0000 37'3 52:57 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:48:13.811063+0000 0 0 2024-04-25T07:21:01.277 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.891189+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:07:11.426953+0000 0 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.892628+0000 37'4 52:68 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:52:07.878867+0000 0 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.897204+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:51:30.387558+0000 0 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892039+0000 37'3 52:71 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:01:00.093189+0000 1 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.895656+0000 37'5 52:73 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:41.312970+0000 2 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892884+0000 37'3 52:71 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:49:26.167660+0000 0 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.894574+0000 37'2 52:70 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:22:07.893974+0000 1 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.895776+0000 37'4 52:68 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:33.686310+0000 0 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:19:46.758210+0000 37'6 52:61 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:15:06.486943+0000 0 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.893616+0000 37'2 52:66 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:51:34.511222+0000 0 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891839+0000 37'5 52:73 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:06.657091+0000 1 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.891804+0000 37'1 52:61 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:08:31.240485+0000 0 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.893188+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:06:33.182195+0000 0 0 2024-04-25T07:21:01.278 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:46.757106+0000 37'2 52:57 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:49:50.049657+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.894954+0000 37'2 52:62 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:14.888489+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.850172+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:53:23.706783+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755995+0000 37'3 52:58 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:27:37.647191+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891582+0000 52'5 52:73 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:16:25.175530+0000 2 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:46.757179+0000 37'4 52:59 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:53:26.802539+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.756164+0000 37'3 52:57 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:16.869210+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892306+0000 37'3 52:63 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:38:24.652869+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.895994+0000 37'1 52:61 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:34:12.123814+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894712+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:23:29.794915+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.756147+0000 37'3 52:58 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:14:48.298332+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894289+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:58:16.590827+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.891035+0000 37'4 52:68 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:06:11.586560+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.893909+0000 37'1 52:61 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:27:15.069777+0000 0 0 2024-04-25T07:21:01.279 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755975+0000 37'3 52:58 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:31.965147+0000 0 0 2024-04-25T07:21:01.280 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.892863+0000 37'2 52:66 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:47:36.332515+0000 0 0 2024-04-25T07:21:01.280 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.896539+0000 37'4 52:72 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:22:04.797910+0000 0 0 2024-04-25T07:21:01.280 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.892753+0000 37'5 52:65 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:11:10.391476+0000 0 0 2024-04-25T07:21:01.280 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:21:01.280 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:01.280 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:21:01.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:21:01.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.1 2024-04-25T07:21:01.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:21:01.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:21:01.295 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:21:01.295 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:01.295 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:01.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.1.asok 2024-04-25T07:21:01.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:21:01.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:21:01.417 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:21:01.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:21:01.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-04-25T07:21:01.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:21:01.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:21:01.432 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:21:01.432 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:01.432 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:01.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.2.asok 2024-04-25T07:21:01.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:21:01.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:21:01.566 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:21:01.579 INFO:tasks.workunit.client.0.smithi102.stdout:revive divergent 0 2024-04-25T07:21:01.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 0' 2024-04-25T07:21:01.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.644633+0000 37'3 52:57 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:48:13.811063+0000 0 0 2024-04-25T07:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.891189+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:07:11.426953+0000 0 0 2024-04-25T07:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.892628+0000 37'4 52:68 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:52:07.878867+0000 0 0 2024-04-25T07:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.897204+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:51:30.387558+0000 0 0 2024-04-25T07:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892039+0000 37'3 52:71 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:01:00.093189+0000 1 0 2024-04-25T07:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.895656+0000 37'5 52:73 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:41.312970+0000 2 0 2024-04-25T07:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892884+0000 37'3 52:71 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:49:26.167660+0000 0 0 2024-04-25T07:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.894574+0000 37'2 52:70 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:22:07.893974+0000 1 0 2024-04-25T07:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.895776+0000 37'4 52:68 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:33.686310+0000 0 0 2024-04-25T07:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:19:46.758210+0000 37'6 52:61 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:15:06.486943+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.893616+0000 37'2 52:66 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:51:34.511222+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891839+0000 37'5 52:73 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:06.657091+0000 1 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.891804+0000 37'1 52:61 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:08:31.240485+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.893188+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:06:33.182195+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:46.757106+0000 37'2 52:57 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:49:50.049657+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.894954+0000 37'2 52:62 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:14.888489+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.850172+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:53:23.706783+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755995+0000 37'3 52:58 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:27:37.647191+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891582+0000 52'5 52:73 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:16:25.175530+0000 2 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:46.757179+0000 37'4 52:59 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:53:26.802539+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.756164+0000 37'3 52:57 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:16.869210+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892306+0000 37'3 52:63 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:38:24.652869+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.895994+0000 37'1 52:61 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:34:12.123814+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894712+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:23:29.794915+0000 0 0 2024-04-25T07:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.756147+0000 37'3 52:58 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:14:48.298332+0000 0 0 2024-04-25T07:21:01.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894289+0000 37'3 52:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:58:16.590827+0000 0 0 2024-04-25T07:21:01.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.891035+0000 37'4 52:68 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:06:11.586560+0000 0 0 2024-04-25T07:21:01.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.893909+0000 37'1 52:61 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:27:15.069777+0000 0 0 2024-04-25T07:21:01.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755975+0000 37'3 52:58 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:31.965147+0000 0 0 2024-04-25T07:21:01.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.892863+0000 37'2 52:66 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:47:36.332515+0000 0 0 2024-04-25T07:21:01.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.896539+0000 37'4 52:72 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:22:04.797910+0000 0 0 2024-04-25T07:21:01.966 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.892753+0000 37'5 52:65 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:11:10.391476+0000 0 0 2024-04-25T07:21:01.966 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:21:01.966 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:01.966 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:21:01.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-04-25T07:21:02.715 INFO:tasks.workunit.client.0.smithi102.stderr:noup is set 2024-04-25T07:21:02.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:21:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:21:02.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:21:02.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:21:02.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:21:02.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:21:02.737 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:21:02.737 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:02.737 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:02.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:21:02.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:21:02.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:21:02.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:21:02.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:21:02.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:21:02.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:21:02.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:21:02.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:21:02.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:21:02.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:21:02.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:21:02.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:21:02.739 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:21:02.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:21:02.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:21:02.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:21:02.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:21:02.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:21:02.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:21:02.772 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:02.774+0000 7fbb06a137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:02.791 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:02.794+0000 7fbb06a137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:02.791 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:02.794+0000 7fbb06a137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:03.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:21:03.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-04-25T07:21:03.332 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:03.334+0000 7fbb06a137c0 -1 Falling back to public interface 2024-04-25T07:21:06.398 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:06.401+0000 7fbb06a137c0 -1 osd.0 37 log_to_monitors true 2024-04-25T07:21:08.196 INFO:tasks.workunit.client.0.smithi102.stdout:delay recovery divergent 2024-04-25T07:21:08.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-04-25T07:21:08.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:08.610 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:08.610 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.644633+0000 37'3 54:60 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:48:13.811063+0000 0 0 2024-04-25T07:21:08.610 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.891189+0000 37'3 53:69 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:07:11.426953+0000 0 0 2024-04-25T07:21:08.610 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.892628+0000 37'4 53:70 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:52:07.878867+0000 0 0 2024-04-25T07:21:08.610 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.897204+0000 37'3 53:69 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:51:30.387558+0000 0 0 2024-04-25T07:21:08.610 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892039+0000 37'3 53:73 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:01:00.093189+0000 1 0 2024-04-25T07:21:08.610 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.895656+0000 37'5 53:75 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:41.312970+0000 2 0 2024-04-25T07:21:08.610 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892884+0000 37'3 53:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:49:26.167660+0000 0 0 2024-04-25T07:21:08.610 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.894574+0000 37'2 53:72 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:22:07.893974+0000 1 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.895776+0000 37'4 53:70 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:33.686310+0000 0 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:19:46.758210+0000 37'6 54:64 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:15:06.486943+0000 0 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.893616+0000 37'2 53:68 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:51:34.511222+0000 0 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891839+0000 37'5 53:75 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:06.657091+0000 1 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.891804+0000 37'1 53:63 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:08:31.240485+0000 0 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.893188+0000 37'3 53:69 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:06:33.182195+0000 0 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:46.757106+0000 37'2 54:60 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:49:50.049657+0000 0 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.894954+0000 37'2 53:64 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:14.888489+0000 0 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.850172+0000 37'3 53:69 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:53:23.706783+0000 0 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755995+0000 37'3 54:61 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:27:37.647191+0000 0 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891582+0000 52'5 53:75 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:16:25.175530+0000 2 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:46.757179+0000 37'4 54:62 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:53:26.802539+0000 0 0 2024-04-25T07:21:08.611 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.756164+0000 37'3 54:60 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:16.869210+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892306+0000 37'3 53:65 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:38:24.652869+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.895994+0000 37'1 53:63 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:34:12.123814+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894712+0000 37'3 53:69 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:23:29.794915+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.756147+0000 37'3 54:61 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:14:48.298332+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894289+0000 37'3 53:69 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:58:16.590827+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.891035+0000 37'4 53:70 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:06:11.586560+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 1 0 0 617 0 0 1 0 1 active+undersized+degraded 2024-04-25T07:19:45.893909+0000 37'1 53:63 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:27:15.069777+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755975+0000 37'3 54:61 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:31.965147+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.892863+0000 37'2 53:68 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:47:36.332515+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.896539+0000 37'4 53:74 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:22:04.797910+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.892753+0000 37'5 53:67 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:11:10.391476+0000 0 0 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:21:08.612 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:08.613 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:21:08.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.0 2024-04-25T07:21:08.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:21:08.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:21:08.626 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:21:08.626 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:08.627 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:08.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.0.asok 2024-04-25T07:21:08.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:21:08.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:21:08.732 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:21:08.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-04-25T07:21:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:noup is unset 2024-04-25T07:21:09.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-04-25T07:21:09.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:09.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:21:09.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:09.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:09.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:09.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:09.495 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:21:09.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:09.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:10.071 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 57 up_thru 57 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6818/3201357640,v1:127.0.0.1:6819/3201357640] [v2:127.0.0.1:6820/3201357640,v1:127.0.0.1:6821/3201357640] exists,up 3d1dc328-65a0-4947-a470-7ad55360f6cb 2024-04-25T07:21:10.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:10.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:10.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:10.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-04-25T07:21:10.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:10.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:21:10.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:10.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:10.072 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:21:10.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:10.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:10.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:10.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:21:10.806 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 44 up_thru 57 down_at 39 last_clean_interval [10,37) [v2:127.0.0.1:6802/550444832,v1:127.0.0.1:6803/550444832] [v2:127.0.0.1:6804/550444832,v1:127.0.0.1:6805/550444832] exists,up df369e44-fe63-4f08-84e4-07f167676b56 2024-04-25T07:21:10.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:10.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:10.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:10.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-04-25T07:21:10.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:10.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:21:10.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:10.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:10.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:10.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:10.807 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:21:10.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:10.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:21:11.254 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 46 up_thru 57 down_at 40 last_clean_interval [15,37) [v2:127.0.0.1:6810/2940722690,v1:127.0.0.1:6811/2940722690] [v2:127.0.0.1:6812/2940722690,v1:127.0.0.1:6813/2940722690] exists,up dcba27a8-55a9-40c9-9cc7-d09a609befd5 2024-04-25T07:21:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:11.666 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:11.666 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:10.576719+0000 37'3 59:64 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:40:35.896146+0000 0 0 2024-04-25T07:21:11.666 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.891189+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:07:11.426953+0000 0 0 2024-04-25T07:21:11.666 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.892628+0000 37'4 55:74 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:52:07.878867+0000 0 0 2024-04-25T07:21:11.666 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.897204+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:51:30.387558+0000 0 0 2024-04-25T07:21:11.666 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892039+0000 37'3 55:77 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:01:00.093189+0000 1 0 2024-04-25T07:21:11.666 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.895656+0000 37'5 55:79 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:41.312970+0000 2 0 2024-04-25T07:21:11.666 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892884+0000 37'3 55:77 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:49:26.167660+0000 0 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.894574+0000 37'2 55:76 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:22:07.893974+0000 1 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.895776+0000 37'4 55:74 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:33.686310+0000 0 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:19:46.758210+0000 37'6 55:66 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:15:06.486943+0000 0 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.893616+0000 37'2 55:72 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:51:34.511222+0000 0 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891839+0000 37'5 55:79 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:06.657091+0000 1 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.576347+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:21:41.910999+0000 0 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.893188+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:06:33.182195+0000 0 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:46.757106+0000 37'2 55:62 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:49:50.049657+0000 0 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:10.574646+0000 37'2 59:63 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:56:46.636872+0000 0 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.850172+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:53:23.706783+0000 0 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755995+0000 37'3 55:63 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:27:37.647191+0000 0 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891582+0000 52'5 55:79 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:16:25.175530+0000 2 0 2024-04-25T07:21:11.667 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:46.757179+0000 37'4 55:64 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:53:26.802539+0000 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:10.577376+0000 37'3 59:64 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:43:55.252807+0000 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 3 1 1 0 0 1851 0 0 3 0 3 active+recovering+degraded 2024-04-25T07:21:10.579530+0000 37'3 59:64 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.575375+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:50:36.505761+0000 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894712+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:23:29.794915+0000 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.756147+0000 37'3 55:63 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:14:48.298332+0000 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894289+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:58:16.590827+0000 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.891035+0000 37'4 55:74 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:06:11.586560+0000 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.573444+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:03:33.635268+0000 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755975+0000 37'3 55:63 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:31.965147+0000 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.892863+0000 37'2 55:72 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:47:36.332515+0000 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.896539+0000 37'4 55:78 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:22:04.797910+0000 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-04-25T07:21:10.576643+0000 37'5 59:65 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:21:11.668 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:11.669 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:21:11.682 INFO:tasks.workunit.client.0.smithi102.stdout:wait for peering 2024-04-25T07:21:11.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-04-25T07:21:11.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:12.119 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:12.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:10.576719+0000 37'3 59:64 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:40:35.896146+0000 0 0 2024-04-25T07:21:12.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.891189+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:07:11.426953+0000 0 0 2024-04-25T07:21:12.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.892628+0000 37'4 55:74 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:52:07.878867+0000 0 0 2024-04-25T07:21:12.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.897204+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:51:30.387558+0000 0 0 2024-04-25T07:21:12.119 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892039+0000 37'3 55:77 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:01:00.093189+0000 1 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.895656+0000 37'5 55:79 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:41.312970+0000 2 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892884+0000 37'3 55:77 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:49:26.167660+0000 0 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.894574+0000 37'2 55:76 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:22:07.893974+0000 1 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.895776+0000 37'4 55:74 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:33.686310+0000 0 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:19:46.758210+0000 37'6 55:66 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:15:06.486943+0000 0 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.893616+0000 37'2 55:72 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:51:34.511222+0000 0 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891839+0000 37'5 55:79 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:06.657091+0000 1 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.576347+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:21:41.910999+0000 0 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.893188+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:06:33.182195+0000 0 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:46.757106+0000 37'2 55:62 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:49:50.049657+0000 0 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:10.574646+0000 37'2 59:63 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:56:46.636872+0000 0 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.850172+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:53:23.706783+0000 0 0 2024-04-25T07:21:12.120 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755995+0000 37'3 55:63 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:27:37.647191+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891582+0000 52'5 55:79 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:16:25.175530+0000 2 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:46.757179+0000 37'4 55:64 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:53:26.802539+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:10.577376+0000 37'3 59:64 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:43:55.252807+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 3 1 1 0 0 1851 0 0 3 0 3 active+recovering+degraded 2024-04-25T07:21:10.579530+0000 37'3 59:64 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.575375+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:50:36.505761+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894712+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:23:29.794915+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.756147+0000 37'3 55:63 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:14:48.298332+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894289+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:58:16.590827+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.891035+0000 37'4 55:74 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:06:11.586560+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.573444+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:03:33.635268+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755975+0000 37'3 55:63 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:31.965147+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.892863+0000 37'2 55:72 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:47:36.332515+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.896539+0000 37'4 55:78 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:22:04.797910+0000 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-04-25T07:21:10.576643+0000 37'5 59:65 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:12.121 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:21:12.122 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:12.122 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:21:12.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-04-25T07:21:12.270 INFO:tasks.workunit.client.0.smithi102.stdout:killing divergent 0 2024-04-25T07:21:12.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 0' 2024-04-25T07:21:12.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:12.709 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:12.710 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:10.576719+0000 37'3 59:64 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:40:35.896146+0000 0 0 2024-04-25T07:21:12.710 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.891189+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:07:11.426953+0000 0 0 2024-04-25T07:21:12.710 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.892628+0000 37'4 55:74 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:52:07.878867+0000 0 0 2024-04-25T07:21:12.710 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.897204+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:51:30.387558+0000 0 0 2024-04-25T07:21:12.710 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892039+0000 37'3 55:77 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:01:00.093189+0000 1 0 2024-04-25T07:21:12.710 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.895656+0000 37'5 55:79 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:41.312970+0000 2 0 2024-04-25T07:21:12.710 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.892884+0000 37'3 55:77 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:49:26.167660+0000 0 0 2024-04-25T07:21:12.710 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.894574+0000 37'2 55:76 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:22:07.893974+0000 1 0 2024-04-25T07:21:12.710 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.895776+0000 37'4 55:74 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:09:33.686310+0000 0 0 2024-04-25T07:21:12.711 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 6 0 0 3702 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:19:46.758210+0000 37'6 55:66 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:15:06.486943+0000 0 0 2024-04-25T07:21:12.711 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.893616+0000 37'2 55:72 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:51:34.511222+0000 0 0 2024-04-25T07:21:12.711 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 5 0 0 3085 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891839+0000 37'5 55:79 [1,2] 1 [1,2] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:08:06.657091+0000 1 0 2024-04-25T07:21:12.711 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.576347+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:21:41.910999+0000 0 0 2024-04-25T07:21:12.711 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.893188+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:06:33.182195+0000 0 0 2024-04-25T07:21:12.711 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:46.757106+0000 37'2 55:62 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:49:50.049657+0000 0 0 2024-04-25T07:21:12.711 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:10.574646+0000 37'2 59:63 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:56:46.636872+0000 0 0 2024-04-25T07:21:12.711 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.850172+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:53:23.706783+0000 0 0 2024-04-25T07:21:12.711 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755995+0000 37'3 55:63 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:27:37.647191+0000 0 0 2024-04-25T07:21:12.711 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 4 0 0 2818 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:19:45.891582+0000 52'5 55:79 [1,2] 1 [1,2] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:16:25.175530+0000 2 0 2024-04-25T07:21:12.711 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:46.757179+0000 37'4 55:64 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:53:26.802539+0000 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:10.577376+0000 37'3 59:64 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:43:55.252807+0000 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 3 1 1 0 0 1851 0 0 3 0 3 active+recovering+degraded 2024-04-25T07:21:10.579530+0000 37'3 59:64 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.575375+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:50:36.505761+0000 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894712+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:23:29.794915+0000 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.756147+0000 37'3 55:63 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:14:48.298332+0000 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:45.894289+0000 37'3 55:73 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:58:16.590827+0000 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.891035+0000 37'4 55:74 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:06:11.586560+0000 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.573444+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:03:33.635268+0000 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 3 0 0 1851 0 0 3 0 3 active+undersized+degraded 2024-04-25T07:19:46.755975+0000 37'3 55:63 [2,1] 2 [2,1] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:31.965147+0000 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 2 0 0 1234 0 0 2 0 2 active+undersized+degraded 2024-04-25T07:19:45.892863+0000 37'2 55:72 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:47:36.332515+0000 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 4 0 0 2468 0 0 4 0 4 active+undersized+degraded 2024-04-25T07:19:45.896539+0000 37'4 55:78 [1,2] 1 [1,2] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:22:04.797910+0000 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-04-25T07:21:10.576643+0000 37'5 59:65 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:21:12.712 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:12.713 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:21:12.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.0 2024-04-25T07:21:12.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:21:12.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:21:12.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:21:12.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:21:12.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:21:12.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:21:12.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.26156.out 2024-04-25T07:21:12.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 0 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.26156.out 2024-04-25T07:21:12.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:21:12.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:21:12.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-25T07:21:12.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:21:12.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-04-25T07:21:12.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.26156.out 2024-04-25T07:21:14.172 INFO:tasks.workunit.client.0.smithi102.stderr:Exporting 1.1b info 1.1b( v 37'5 lc 0'0 (0'0,37'5] local-lis/les=57/58 n=5 ec=33/23 lis/c=57/35 les/c/f=58/37/0 sis=57) 2024-04-25T07:21:14.180 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:21:14.191 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-04-25T07:21:14.205 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dd677bd3:::existing_35:head# 2024-04-25T07:21:14.223 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-04-25T07:21:14.250 INFO:tasks.workunit.client.0.smithi102.stderr:Export successful 2024-04-25T07:21:14.251 INFO:tasks.workunit.client.0.smithi102.stdout: marking collection for removal 2024-04-25T07:21:14.251 INFO:tasks.workunit.client.0.smithi102.stdout:setting '_remove' omap key 2024-04-25T07:21:14.251 INFO:tasks.workunit.client.0.smithi102.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-04-25T07:21:14.372 INFO:tasks.workunit.client.0.smithi102.stderr:Remove successful 2024-04-25T07:21:14.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 0 --op import --file td/divergent-priors/exp.26156.out 2024-04-25T07:21:14.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:21:14.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:21:14.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-25T07:21:14.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:21:14.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-04-25T07:21:14.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op import --file td/divergent-priors/exp.26156.out 2024-04-25T07:21:15.846 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e59 pg_nums {1={23=1,33=32}} deleted_pools ) 2024-04-25T07:21:15.846 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.1b 2024-04-25T07:21:15.846 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:21:15.846 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:21:15.925 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-04-25T07:21:15.925 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:21:15.967 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dd677bd3:::existing_35:head# 2024-04-25T07:21:15.967 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:21:16.009 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-04-25T07:21:16.009 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T07:21:16.050 INFO:tasks.workunit.client.0.smithi102.stdout:write_pg epoch 58 info 1.1b( v 37'5 lc 0'0 (0'0,37'5] local-lis/les=57/58 n=5 ec=33/23 lis/c=57/35 les/c/f=58/37/0 sis=57) 2024-04-25T07:21:16.092 INFO:tasks.workunit.client.0.smithi102.stdout:Import successful 2024-04-25T07:21:16.640 INFO:tasks.workunit.client.0.smithi102.stdout:reviving divergent 0 2024-04-25T07:21:16.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 0' 2024-04-25T07:21:16.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:17.061 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:17.061 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:10.576719+0000 37'3 59:64 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:40:35.896146+0000 0 0 2024-04-25T07:21:17.061 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:11.014767+0000 37'3 59:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:18:36.945342+0000 0 0 2024-04-25T07:21:17.061 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:11.127523+0000 37'4 59:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:13:12.364212+0000 0 0 2024-04-25T07:21:17.061 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:11.123135+0000 37'3 59:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:31:10.998409+0000 0 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:11.122396+0000 37'3 59:90 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:22:43.636142+0000 1 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-04-25T07:21:11.129012+0000 37'5 59:91 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 no scrub is scheduled 2 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:11.012388+0000 37'3 59:90 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:34:45.482914+0000 0 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:11.121651+0000 37'2 59:89 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:37:47.522627+0000 1 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-04-25T07:21:11.013789+0000 59'5 59:88 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:18:02.558796+0000 0 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-04-25T07:21:10.468029+0000 37'6 59:79 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:48:35.842526+0000 0 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:11.128028+0000 37'2 59:85 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:13:07.372825+0000 0 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-04-25T07:21:11.012096+0000 37'5 59:92 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:14:54.549459+0000 1 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.576347+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:21:41.910999+0000 0 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:11.015702+0000 37'3 59:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:19:18.765000+0000 0 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:10.574577+0000 37'2 59:75 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:30:59.412314+0000 0 0 2024-04-25T07:21:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:10.574646+0000 37'2 59:63 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:56:46.636872+0000 0 0 2024-04-25T07:21:17.063 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:11.014022+0000 37'3 59:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:07:10.018812+0000 0 0 2024-04-25T07:21:17.063 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:10.465332+0000 37'3 59:76 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:39:44.030303+0000 0 0 2024-04-25T07:21:17.063 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovery_wait+degraded 2024-04-25T07:21:11.127011+0000 52'5 59:91 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 2 0 2024-04-25T07:21:17.063 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:10.467499+0000 37'4 59:77 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:43:54.401941+0000 0 0 2024-04-25T07:21:17.063 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:10.577376+0000 37'3 59:64 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:43:55.252807+0000 0 0 2024-04-25T07:21:17.063 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 3 1 1 0 0 1851 0 0 3 0 3 active+recovering+degraded 2024-04-25T07:21:10.579530+0000 37'3 59:64 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:17.063 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.575375+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:50:36.505761+0000 0 0 2024-04-25T07:21:17.063 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:11.128582+0000 37'3 59:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:31:44.837139+0000 0 0 2024-04-25T07:21:17.063 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:10.466938+0000 37'3 59:76 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:41:35.193593+0000 0 0 2024-04-25T07:21:17.063 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:11.013124+0000 37'3 59:86 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:46:00.917594+0000 0 0 2024-04-25T07:21:17.064 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:11.015110+0000 37'4 59:87 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:16:23.123152+0000 0 0 2024-04-25T07:21:17.064 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:10.573444+0000 37'1 59:62 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:03:33.635268+0000 0 0 2024-04-25T07:21:17.064 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:10.465152+0000 37'3 59:76 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:15:05.227466+0000 0 0 2024-04-25T07:21:17.064 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:11.123169+0000 37'2 59:85 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:01:55.333390+0000 0 0 2024-04-25T07:21:17.064 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:11.122314+0000 37'4 59:91 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:21:55.756679+0000 0 0 2024-04-25T07:21:17.064 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-04-25T07:21:10.576643+0000 37'5 59:65 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:17.064 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:21:17.064 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:17.064 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:21:17.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:21:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:21:17.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:21:17.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:21:17.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:21:17.081 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:21:17.081 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:17.081 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:17.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:21:17.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:21:17.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:21:17.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:21:17.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:21:17.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:21:17.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:21:17.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:21:17.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:21:17.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:21:17.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:21:17.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:21:17.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:21:17.084 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:21:17.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:21:17.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:21:17.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:21:17.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:21:17.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:21:17.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:21:17.122 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:17.125+0000 7f296b0c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:17.124 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:17.125+0000 7f296b0c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:17.126 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:17.129+0000 7f296b0c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:17.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:21:17.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:17.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:21:17.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:17.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:17.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:17.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:17.491 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:21:17.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:17.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:17.695 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:17.697+0000 7f296b0c57c0 -1 Falling back to public interface 2024-04-25T07:21:17.953 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 57 up_thru 57 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6818/3201357640,v1:127.0.0.1:6819/3201357640] [v2:127.0.0.1:6820/3201357640,v1:127.0.0.1:6821/3201357640] exists,up 3d1dc328-65a0-4947-a470-7ad55360f6cb 2024-04-25T07:21:17.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:17.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:17.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:17.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 0 2024-04-25T07:21:17.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:17.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:21:17.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:17.955 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:21:17.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:17.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:17.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:17.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:17.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:18.361 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 57 up_thru 57 down_at 38 last_clean_interval [5,37) [v2:127.0.0.1:6818/3201357640,v1:127.0.0.1:6819/3201357640] [v2:127.0.0.1:6820/3201357640,v1:127.0.0.1:6821/3201357640] exists,up 3d1dc328-65a0-4947-a470-7ad55360f6cb 2024-04-25T07:21:18.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:18.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:18.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:18.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-04-25T07:21:20.769 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:20.769+0000 7f296b0c57c0 -1 osd.0 59 log_to_monitors true 2024-04-25T07:21:38.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.0 2024-04-25T07:21:38.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:21:38.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:21:38.365 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:21:38.365 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:38.365 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:38.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.0.asok 2024-04-25T07:21:38.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:21:38.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.0.asok dump_ops_in_flight 2024-04-25T07:21:38.482 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:21:38.482 INFO:tasks.workunit.client.0.smithi102.stdout: "ops": [], 2024-04-25T07:21:38.482 INFO:tasks.workunit.client.0.smithi102.stdout: "num_ops": 0 2024-04-25T07:21:38.482 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:21:38.494 INFO:tasks.workunit.client.0.smithi102.stdout:allowing recovery 2024-04-25T07:21:38.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-04-25T07:21:38.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:38.911 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:38.911 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.820184+0000 37'3 64:74 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:58:15.123801+0000 0 0 2024-04-25T07:21:38.911 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.822158+0000 37'3 64:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:03:57.915879+0000 0 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:22.820725+0000 37'4 64:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:36:47.740033+0000 0 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.820131+0000 37'3 64:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:30:47.680208+0000 0 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.822777+0000 37'3 64:103 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:43:31.372328+0000 1 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-04-25T07:21:22.822337+0000 37'5 64:103 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 no scrub is scheduled 2 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.815019+0000 37'3 64:103 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:12:04.648909+0000 0 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:22.818519+0000 37'2 64:102 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:26:51.702542+0000 1 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-04-25T07:21:22.818825+0000 59'5 64:101 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:25:08.110713+0000 0 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-04-25T07:21:22.824425+0000 37'6 64:92 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:04:36.467368+0000 0 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:22.821382+0000 37'2 64:98 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:44:16.613897+0000 0 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-04-25T07:21:22.812511+0000 37'5 64:105 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:19:24.108025+0000 1 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:22.819699+0000 37'1 64:72 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:50:38.730883+0000 0 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.817919+0000 37'3 64:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:58.074169+0000 0 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:22.821240+0000 37'2 64:88 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:50:12.895914+0000 0 0 2024-04-25T07:21:38.912 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:22.829363+0000 37'2 64:73 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:43:47.278600+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.814893+0000 37'3 64:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:16:57.016954+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.816946+0000 37'3 64:89 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:56:23.369052+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovering+degraded 2024-04-25T07:21:22.829752+0000 52'5 64:104 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 2 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:22.819555+0000 37'4 64:90 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:30:11.915652+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.823778+0000 37'3 64:74 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:16:16.952869+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 3 1 1 0 0 1851 0 0 3 0 3 active+recovery_wait+degraded 2024-04-25T07:21:22.822893+0000 37'3 64:72 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:22.829857+0000 37'1 64:72 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:11:04.498476+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.821973+0000 37'3 64:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:46:29.484592+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.742698+0000 37'3 64:89 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:06:10.083212+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.812502+0000 37'3 64:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:29:59.419468+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:22.814591+0000 37'4 64:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:17:58.281658+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:22.828236+0000 37'1 64:72 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:02:57.482755+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.820764+0000 37'3 64:89 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:20:27.560860+0000 0 0 2024-04-25T07:21:38.913 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:22.823386+0000 37'2 64:98 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:34:57.691798+0000 0 0 2024-04-25T07:21:38.914 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:22.819431+0000 37'4 64:104 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:48:21.916282+0000 0 0 2024-04-25T07:21:38.914 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-04-25T07:21:22.825119+0000 37'5 64:74 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:38.914 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:21:38.914 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:38.914 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:21:38.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:21:38.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:21:39.079 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:21:39.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:21:39.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:21:39.268 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:21:39.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:21:39.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:21:39.431 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:21:39.444 INFO:tasks.workunit.client.0.smithi102.stdout:reading divergent objects 2024-04-25T07:21:39.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-04-25T07:21:39.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:39.841 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:39.841 INFO:tasks.workunit.client.0.smithi102.stdout:1.1f 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.820184+0000 37'3 64:74 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:49.603722+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:58:15.123801+0000 0 0 2024-04-25T07:21:39.841 INFO:tasks.workunit.client.0.smithi102.stdout:1.1e 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.822158+0000 37'3 64:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:46.720661+0000 0'0 2024-04-25T07:18:46.720661+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:03:57.915879+0000 0 0 2024-04-25T07:21:39.841 INFO:tasks.workunit.client.0.smithi102.stdout:1.1d 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:22.820725+0000 37'4 64:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:44.730612+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:36:47.740033+0000 0 0 2024-04-25T07:21:39.841 INFO:tasks.workunit.client.0.smithi102.stdout:1.b 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.820131+0000 37'3 64:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:51.727922+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:30:47.680208+0000 0 0 2024-04-25T07:21:39.841 INFO:tasks.workunit.client.0.smithi102.stdout:1.a 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.822777+0000 37'3 64:103 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:56.743609+0000 0'0 2024-04-25T07:18:29.457836+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:43:31.372328+0000 1 0 2024-04-25T07:21:39.841 INFO:tasks.workunit.client.0.smithi102.stdout:1.9 5 0 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-04-25T07:21:22.822337+0000 37'5 64:103 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:58.780769+0000 37'2 2024-04-25T07:18:58.780769+0000 0 1 no scrub is scheduled 2 0 2024-04-25T07:21:39.841 INFO:tasks.workunit.client.0.smithi102.stdout:1.8 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.815019+0000 37'3 64:103 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:57.793145+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:12:04.648909+0000 0 0 2024-04-25T07:21:39.841 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:22.818519+0000 37'2 64:102 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:19:00.907823+0000 37'1 2024-04-25T07:19:00.907823+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:26:51.702542+0000 1 0 2024-04-25T07:21:39.841 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-04-25T07:21:22.818825+0000 59'5 64:101 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:53.803633+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:25:08.110713+0000 0 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 6 0 0 0 0 3702 0 0 6 0 6 active+clean 2024-04-25T07:21:22.824425+0000 37'6 64:92 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:49.854093+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:04:36.467368+0000 0 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:22.821382+0000 37'2 64:98 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:50.740362+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:44:16.613897+0000 0 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 5 0 0 0 0 3085 0 0 5 0 5 active+clean 2024-04-25T07:21:22.812511+0000 37'5 64:105 [1,2,0] 1 [1,2,0] 1 37'1 2024-04-25T07:18:55.832649+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:19:24.108025+0000 1 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:22.819699+0000 37'1 64:72 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:46.514785+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:50:38.730883+0000 0 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.c 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.817919+0000 37'3 64:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:49.751537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:42:58.074169+0000 0 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:22.821240+0000 37'2 64:88 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:46.815352+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:50:12.895914+0000 0 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.1c 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:22.829363+0000 37'2 64:73 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:53.669856+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:43:47.278600+0000 0 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.814893+0000 37'3 64:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:43.669427+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:16:57.016954+0000 0 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.d 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.816946+0000 37'3 64:89 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:47.844530+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:56:23.369052+0000 0 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.e 4 0 1 0 0 2818 0 0 5 0 5 active+recovering+degraded 2024-04-25T07:21:22.829752+0000 52'5 64:104 [1,2,0] 1 [1,2,0] 1 37'2 2024-04-25T07:18:59.846929+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 2 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.f 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:22.819555+0000 37'4 64:90 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:48.890915+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:30:11.915652+0000 0 0 2024-04-25T07:21:39.842 INFO:tasks.workunit.client.0.smithi102.stdout:1.10 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.823778+0000 37'3 64:74 [0,2,1] 0 [0,2,1] 0 0'0 2024-04-25T07:18:48.590830+0000 0'0 2024-04-25T07:18:48.590830+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:16:16.952869+0000 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:1.11 3 1 1 0 0 1851 0 0 3 0 3 active+recovery_wait+degraded 2024-04-25T07:21:22.822893+0000 37'3 64:72 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:47.552607+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:1.12 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:22.829857+0000 37'1 64:72 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:52.640360+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:11:04.498476+0000 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:1.13 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.821973+0000 37'3 64:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:45.842558+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:46:29.484592+0000 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:1.14 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.742698+0000 37'3 64:89 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T07:18:50.874992+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:06:10.083212+0000 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:1.15 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.812502+0000 37'3 64:99 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:52.731797+0000 0'0 2024-04-25T07:18:52.731797+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:29:59.419468+0000 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:1.16 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:22.814591+0000 37'4 64:100 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:48.724543+0000 0'0 2024-04-25T07:18:48.724543+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:17:58.281658+0000 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:1.17 1 0 0 0 0 617 0 0 1 0 1 active+clean 2024-04-25T07:21:22.828236+0000 37'1 64:72 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:51.601531+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:02:57.482755+0000 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:1.18 3 0 0 0 0 1851 0 0 3 0 3 active+clean 2024-04-25T07:21:22.820764+0000 37'3 64:89 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T07:18:45.840783+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:20:27.560860+0000 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:1.19 2 0 0 0 0 1234 0 0 2 0 2 active+clean 2024-04-25T07:21:22.823386+0000 37'2 64:98 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:18:47.700378+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:34:57.691798+0000 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:1.1a 4 0 0 0 0 2468 0 0 4 0 4 active+clean 2024-04-25T07:21:22.819431+0000 37'4 64:104 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T07:18:54.759537+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:48:21.916282+0000 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:1.1b 5 1 1 0 0 3085 0 0 5 0 5 active+recovery_wait+degraded 2024-04-25T07:21:22.825119+0000 37'5 64:74 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T07:18:50.605734+0000 0'0 2024-04-25T07:18:29.457836+0000 0 1 no scrub is scheduled 0 0 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:39.843 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:21:39.857 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-04-25T07:21:39.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-04-25T07:21:39.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:21:39.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:21:39.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:21:39.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:21:39.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:21:39.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:21:39.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:21:39.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:21:39.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-04-25T07:21:39.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:21:39.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-04-25T07:21:40.027 INFO:tasks.workunit.client.0.smithi102.stderr:error getting test/newobject_1: (2) No such file or directory 2024-04-25T07:21:40.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:21:40.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-04-25T07:21:40.062 INFO:tasks.workunit.client.0.smithi102.stderr:error getting test/newobject_2: (2) No such file or directory 2024-04-25T07:21:40.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-04-25T07:21:40.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:21:40.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:21:40.082 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4674.0:1 2024-04-25T07:19:19.594852+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:21:40.082 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 20 _merge_object_divergent_entries: keeping 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4674.0:1 2024-04-25T07:19:19.594852+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:21:40.082 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:21:40.082 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 37'1 first_divergent_update: 37'6 last_divergent_update: 37'6 2024-04-25T07:21:40.082 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-04-25T07:21:40.082 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:21:40.082 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 37'6 2024-04-25T07:21:40.082 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4674.0:1 2024-04-25T07:19:19.594852+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:21:40.082 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-04-25T07:21:40.083 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 37'6 (37'1) delete 1:70b3bf12:::existing_4:head by client.4680.0:1 2024-04-25T07:19:19.604640+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:21:40.083 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 20 _merge_object_divergent_entries: keeping 37'6 (37'1) delete 1:70b3bf12:::existing_4:head by client.4680.0:1 2024-04-25T07:19:19.604640+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:21:40.083 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:21:40.083 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 37'1 first_divergent_update: 37'6 last_divergent_update: 37'6 2024-04-25T07:21:40.083 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head has no more recent entries in log 2024-04-25T07:21:40.083 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:21:40.083 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head olog_can_rollback_to: 37'6 2024-04-25T07:21:40.083 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head cannot rollback 37'6 (37'1) delete 1:70b3bf12:::existing_4:head by client.4680.0:1 2024-04-25T07:19:19.604640+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head cannot roll back, removing and adding to missing 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 37'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4677.0:1 2024-04-25T07:19:19.600625+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 20 _merge_object_divergent_entries: keeping 37'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4677.0:1 2024-04-25T07:19:19.600625+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 37'5 last_divergent_update: 37'5 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:10.089+0000 7f3d78bf8640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 37'4 (37'1) delete 1:8881e6d8:::existing_3:head by client.4685.0:1 2024-04-25T07:19:19.611085+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 20 _merge_object_divergent_entries: keeping 37'4 (37'1) delete 1:8881e6d8:::existing_3:head by client.4685.0:1 2024-04-25T07:19:19.611085+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 37'1 first_divergent_update: 37'4 last_divergent_update: 37'4 2024-04-25T07:21:40.089 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 37'4 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 37'4 (37'1) delete 1:8881e6d8:::existing_3:head by client.4685.0:1 2024-04-25T07:19:19.611085+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 37'7 (37'1) modify 1:d858c035:::existing_1:head by client.4682.0:1 2024-04-25T07:19:19.608041+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 20 _merge_object_divergent_entries: keeping 37'7 (37'1) modify 1:d858c035:::existing_1:head by client.4682.0:1 2024-04-25T07:19:19.608041+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 37'1 first_divergent_update: 37'7 last_divergent_update: 37'7 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 37'7 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 37'7 (37'1) modify 1:d858c035:::existing_1:head by client.4682.0:1 2024-04-25T07:19:19.608041+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-04-25T07:21:40.090 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 37'6 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4670.0:1 2024-04-25T07:19:19.586936+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:21:40.091 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 20 _merge_object_divergent_entries: keeping 37'6 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4670.0:1 2024-04-25T07:19:19.586936+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:21:40.091 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:21:40.091 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 37'6 last_divergent_update: 37'6 2024-04-25T07:21:40.091 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-04-25T07:21:40.091 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.089+0000 7fbae999b640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-04-25T07:21:40.091 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.937+0000 7fbae8999640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 37'6 (37'1) delete 1:70b3bf12:::existing_4:head by client.4680.0:1 2024-04-25T07:19:19.604640+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:21:40.091 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.937+0000 7fbae8999640 20 _merge_object_divergent_entries: keeping 37'6 (37'1) delete 1:70b3bf12:::existing_4:head by client.4680.0:1 2024-04-25T07:19:19.604640+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:21:40.091 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.937+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.937+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 37'1 first_divergent_update: 37'6 last_divergent_update: 37'6 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.937+0000 7fbae8999640 10 _merge_object_divergent_entries: more recent entry found: 52'5 (37'1) modify 1:70b3bf12:::existing_4:head by client.4802.0:1 2024-04-25T07:20:24.316151+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.937+0000 7fbae8999640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 37'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4677.0:1 2024-04-25T07:19:19.600625+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.937+0000 7fbae8999640 20 _merge_object_divergent_entries: keeping 37'5 (0'0) modify 1:75ac5091:::newobject_2:head by client.4677.0:1 2024-04-25T07:19:19.600625+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.937+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.937+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 37'5 last_divergent_update: 37'5 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.937+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.937+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.941+0000 7fbae8999640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4674.0:1 2024-04-25T07:19:19.594852+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.941+0000 7fbae8999640 20 _merge_object_divergent_entries: keeping 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4674.0:1 2024-04-25T07:19:19.594852+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.941+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.941+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 37'1 first_divergent_update: 37'6 last_divergent_update: 37'6 2024-04-25T07:21:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.941+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-04-25T07:21:40.093 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.941+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:21:40.093 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.941+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 37'6 2024-04-25T07:21:40.093 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.941+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 37'6 (37'1) modify 1:97db2a0f:::existing_2:head by client.4674.0:1 2024-04-25T07:19:19.594852+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:21:40.093 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:10.941+0000 7fbae8999640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-04-25T07:21:40.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:21:40.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:21:40.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:21:40.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:21:40.129 INFO:tasks.workunit.client.0.smithi102.stdout:success 2024-04-25T07:21:40.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-04-25T07:21:40.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.26156.out 2024-04-25T07:21:40.130 INFO:tasks.workunit.client.0.smithi102.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.26156.out': No such file or directory 2024-04-25T07:21:40.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-04-25T07:21:40.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:21:40.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:21:40.665 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T07:21:40.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-04-25T07:21:40.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:21:40.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:21:40.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:21:40.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:21:40.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:21:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:21:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:21:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:21:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:21:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:21:40.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:21:40.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:21:40.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:21:40.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:21:40.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:21:41.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:21:41.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:21:41.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:21:41.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:21:41.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:21:41.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:21:41.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:21:41.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:21:41.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:21:41.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:21:41.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:21:41.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:21:41.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:21:41.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:21:41.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:21:41.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:41.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:41.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26156 2024-04-25T07:21:41.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:21:41.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:21:41.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:21:41.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:21:41.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:21:41.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:21:41.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:21:41.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:21:41.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:21:41.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:21:41.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:21:41.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:21:41.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:21:41.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:21:41.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:21:41.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:21:41.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:21:41.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:21:41.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:21:41.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:21:41.045 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:21:41.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:21:41.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:21:41.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:21:41.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:21:41.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:21:41.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:21:41.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:21:41.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:21:41.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:41.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:41.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26156 2024-04-25T07:21:41.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:21:41.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:21:41.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:21:41.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:21:41.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:41.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:41.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26156 2024-04-25T07:21:41.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:21:41.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:21:41.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:21:41.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:21:41.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-04-25T07:21:41.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-04-25T07:21:41.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-04-25T07:21:41.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-04-25T07:21:41.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-04-25T07:21:41.054 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-04-25T07:21:41.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-04-25T07:21:41.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-04-25T07:21:41.056 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:21:41.056 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:21:41.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-04-25T07:21:41.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:21:41.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:21:41.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:21:41.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:21:41.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:21:41.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:21:41.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:21:41.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:21:41.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:21:41.580 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:21:41.581 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:41.581 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:41.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:21:41.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:21:41.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:21:41.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:21:41.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:21:41.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:21:41.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:21:41.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:21:41.818 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:21:41.818 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:21:41.818 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:21:41.819 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:21:41.819 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:41.819 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:41.819 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-mon.a.asok 2024-04-25T07:21:41.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:21:41.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26156/ceph-mon.a.asok config get fsid 2024-04-25T07:21:41.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:21:41.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:21:41.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:21:41.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:21:41.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:21:41.936 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:21:41.936 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:21:41.936 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:21:41.937 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:21:41.937 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:41.937 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:41.937 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-mon.a.asok 2024-04-25T07:21:41.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:21:41.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26156/ceph-mon.a.asok config get mon_host 2024-04-25T07:21:42.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-04-25T07:21:42.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:21:42.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:21:42.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:21:42.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:21:42.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:21:42.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:21:42.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:21:42.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:21:42.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:21:42.973 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:21:42.973 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:42.973 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:42.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:21:42.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:21:42.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:21:43.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:21:43.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-04-25T07:21:43.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:21:43.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:21:43.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:21:43.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:21:43.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:21:43.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:21:43.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:21:43.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:21:43.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:21:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:21:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:21:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:21:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:21:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:21:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:21:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:21:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:21:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:21:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:21:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:21:43.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:21:43.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=87e4e2ff-6f89-4a5e-9a0b-7ee1f6361e7e 2024-04-25T07:21:43.051 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 87e4e2ff-6f89-4a5e-9a0b-7ee1f6361e7e 2024-04-25T07:21:43.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 87e4e2ff-6f89-4a5e-9a0b-7ee1f6361e7e' 2024-04-25T07:21:43.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:21:43.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHBCpmeiGvBBAATSHkQhxZZI/AAtjPvwOWCw== 2024-04-25T07:21:43.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHBCpmeiGvBBAATSHkQhxZZI/AAtjPvwOWCw=="}' 2024-04-25T07:21:43.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 87e4e2ff-6f89-4a5e-9a0b-7ee1f6361e7e -i td/divergent-priors/0/new.json 2024-04-25T07:21:43.403 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:21:43.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:21:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHBCpmeiGvBBAATSHkQhxZZI/AAtjPvwOWCw== --osd-uuid 87e4e2ff-6f89-4a5e-9a0b-7ee1f6361e7e 2024-04-25T07:21:43.469 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:43.469+0000 7fa1134337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:43.712 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:43.481+0000 7fa1134337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:43.712 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:43.481+0000 7fa1134337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:43.712 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:43.481+0000 7fa1134337c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:21:43.712 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:43.481+0000 7fa1134337c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:21:43.712 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:43.481+0000 7fa1134337c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:21:43.712 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:43.513+0000 7fa1134337c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:21:49.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:21:49.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:21:49.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:21:49.878 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:21:49.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:21:50.463 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:21:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:21:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:21:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:21:50.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:21:50.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:21:50.502 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:50.504+0000 7f079a30c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:50.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:50.512+0000 7f079a30c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:50.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:50.512+0000 7f079a30c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:50.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:21:50.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:50.893 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:21:50.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:21:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:50.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:51.075 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:51.076+0000 7f079a30c7c0 -1 Falling back to public interface 2024-04-25T07:21:51.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:21:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:21:52.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:52.320 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:21:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:21:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:52.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:52.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:21:53.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:21:53.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:53.742 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:21:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:21:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:21:53.760+0000 7f079a30c7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:21:54.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:21:55.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:21:55.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:55.256 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:21:55.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:21:55.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:55.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:55.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:21:56.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:21:56.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:56.740 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:21:56.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:21:56.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:56.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:57.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:21:58.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:21:58.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:58.207 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:21:58.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:21:58.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:58.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:58.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:21:59.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:21:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:59.676 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:21:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:21:59.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:59.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:00.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:01.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:01.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:01.143 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:22:01.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:22:01.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:01.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:01.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:02.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:02.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:02.602 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:22:02.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:22:02.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:02.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:03.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:04.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:04.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:04.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:22:04.046 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:22:04.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:04.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:04.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:05.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:05.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:05.609 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:22:05.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:22:05.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:05.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:06.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:07.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:07.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:07.050 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:22:07.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:22:07.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:07.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:07.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:08.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:08.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:08.663 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:22:08.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:22:08.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:08.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:09.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:10.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:10.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:10.446 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:22:10.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:22:10.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:10.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:11.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:12.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:12.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:12.448 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:22:12.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:22:12.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:12.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:13.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:14.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:14.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:14.294 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:22:14.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:22:14.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:14.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:14.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:15.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:15.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:15.909 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:22:15.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:22:15.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:15.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:16.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:17.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:17.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:17.953 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:22:17.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:22:17.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:17.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:18.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:19.421 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:22:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:22:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:19.848 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2999561753,v1:127.0.0.1:6803/2999561753] [v2:127.0.0.1:6804/2999561753,v1:127.0.0.1:6805/2999561753] exists,up 87e4e2ff-6f89-4a5e-9a0b-7ee1f6361e7e 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:22:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:22:19.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:22:19.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:22:19.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7e6be8e7-02ca-4c7c-a7ae-17fd198b434f 2024-04-25T07:22:19.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7e6be8e7-02ca-4c7c-a7ae-17fd198b434f' 2024-04-25T07:22:19.856 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 7e6be8e7-02ca-4c7c-a7ae-17fd198b434f 2024-04-25T07:22:19.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:22:19.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrBCpmrawCNRAAjgfY3wybh3EiOfynHaU8jg== 2024-04-25T07:22:19.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrBCpmrawCNRAAjgfY3wybh3EiOfynHaU8jg=="}' 2024-04-25T07:22:19.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7e6be8e7-02ca-4c7c-a7ae-17fd198b434f -i td/divergent-priors/1/new.json 2024-04-25T07:22:20.467 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:22:20.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:22:20.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrBCpmrawCNRAAjgfY3wybh3EiOfynHaU8jg== --osd-uuid 7e6be8e7-02ca-4c7c-a7ae-17fd198b434f 2024-04-25T07:22:20.526 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:20.528+0000 7fdc584407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:20.552 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:20.552+0000 7fdc584407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:20.553 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:20.552+0000 7fdc584407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:20.554 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:20.556+0000 7fdc584407c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:22:20.554 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:20.556+0000 7fdc584407c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:22:20.554 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:20.556+0000 7fdc584407c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:22:20.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:20.592+0000 7fdc584407c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:22:27.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:22:27.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:22:27.357 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:22:27.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:22:27.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:22:27.942 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:22:27.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:22:27.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:22:27.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:22:27.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:22:27.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:22:27.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:27.980+0000 7f466ca5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:28.015 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:28.016+0000 7f466ca5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:28.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:28.016+0000 7f466ca5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:28.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:22:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:22:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:22:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:22:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:22:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:28.389 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:22:28.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:22:28.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:28.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:28.591 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:28.592+0000 7f466ca5c7c0 -1 Falling back to public interface 2024-04-25T07:22:28.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:29.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:29.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:29.808 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:22:29.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:22:29.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:29.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:30.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:31.165 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:31.164+0000 7f466ca5c7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:22:31.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:31.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:31.255 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:22:31.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:22:31.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:31.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:32.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:33.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:33.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:33.009 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:22:33.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:22:33.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:33.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:33.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:34.528 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:22:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:22:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:35.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:36.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:36.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:36.105 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:22:36.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:22:36.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:36.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:36.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:37.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:37.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:37.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:22:37.562 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:22:37.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:37.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:38.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:39.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:39.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:39.079 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:22:39.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:22:39.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:39.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:39.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:40.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:40.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:40.531 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:22:40.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:22:40.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:40.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:41.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:42.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:42.090 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:22:42.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:22:42.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:42.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:42.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:43.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:43.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:43.568 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:22:43.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:22:43.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:43.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:44.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:45.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:45.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:45.096 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:22:45.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:22:45.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:45.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:45.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:46.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:46.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:46.641 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:22:46.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:22:46.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:46.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:47.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:48.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:48.476 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:22:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:22:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:48.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:49.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:50.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:22:50.429 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:22:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:51.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:52.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:52.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:52.302 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:22:52.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:22:52.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:52.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:53.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:54.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:54.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:54.082 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:22:54.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:22:54.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:54.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:54.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:55.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:55.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:55.848 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:22:55.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:22:55.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:55.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:56.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:57.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:57.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:57.380 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:22:57.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:22:57.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:57.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:22:57.821 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2403496704,v1:127.0.0.1:6811/2403496704] [v2:127.0.0.1:6812/2403496704,v1:127.0.0.1:6813/2403496704] exists,up 7e6be8e7-02ca-4c7c-a7ae-17fd198b434f 2024-04-25T07:22:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:22:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:22:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:22:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:22:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-04-25T07:22:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:22:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:22:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:57.826 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:57.827 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:57.827 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:22:57.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:22:57.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:22:57.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:22:57.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:22:57.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:22:57.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:22:57.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:22:57.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:22:57.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:22:57.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:22:57.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:22:57.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:22:57.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:22:57.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:22:57.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:22:57.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58d268a7-eec2-4e03-8fee-7633b0a83ab6 2024-04-25T07:22:57.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 58d268a7-eec2-4e03-8fee-7633b0a83ab6' 2024-04-25T07:22:57.831 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 58d268a7-eec2-4e03-8fee-7633b0a83ab6 2024-04-25T07:22:57.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:22:57.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRBCpmBo94MxAAlxew0c0P1J2e8kuFAnn47w== 2024-04-25T07:22:57.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRBCpmBo94MxAAlxew0c0P1J2e8kuFAnn47w=="}' 2024-04-25T07:22:57.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58d268a7-eec2-4e03-8fee-7633b0a83ab6 -i td/divergent-priors/2/new.json 2024-04-25T07:22:58.417 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:22:58.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:22:58.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRBCpmBo94MxAAlxew0c0P1J2e8kuFAnn47w== --osd-uuid 58d268a7-eec2-4e03-8fee-7633b0a83ab6 2024-04-25T07:22:58.479 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:58.483+0000 7f6bc88b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:58.481 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:58.483+0000 7f6bc88b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:58.482 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:58.487+0000 7f6bc88b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:58.483 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:58.487+0000 7f6bc88b87c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:22:58.483 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:58.487+0000 7f6bc88b87c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:22:58.483 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:58.487+0000 7f6bc88b87c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:22:58.518 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:22:58.523+0000 7f6bc88b87c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:23:05.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:23:05.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:23:05.108 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:23:05.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:23:05.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:23:05.673 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:23:05.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:23:05.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:23:05.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:23:05.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:23:05.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:23:05.714 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:23:05.719+0000 7fec899637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:05.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:23:05.747+0000 7fec899637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:05.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:23:05.747+0000 7fec899637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:06.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:23:06.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:23:06.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:23:06.140 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:23:06.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:23:06.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:23:06.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:06.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:23:06.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:06.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:06.315 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:23:06.319+0000 7fec899637c0 -1 Falling back to public interface 2024-04-25T07:23:06.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:07.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:07.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:07.659 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:23:07.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:23:07.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:07.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:08.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:08.851 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:23:08.855+0000 7fec899637c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:23:09.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:09.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:09.091 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:23:09.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:23:09.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:09.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:09.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:10.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:10.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:10.719 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:23:10.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:23:10.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:10.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:11.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:12.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:12.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:12.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:23:12.199 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:23:12.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:12.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:12.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:13.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:13.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:13.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:23:13.772 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:23:13.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:13.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:14.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:15.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:15.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:15.195 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:23:15.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:23:15.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:15.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:15.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:16.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:16.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:16.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:23:16.846 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:23:16.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:16.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:17.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:18.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:18.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:18.285 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:23:18.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:23:18.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:18.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:18.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:19.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:19.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:19.788 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:23:19.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:23:19.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:19.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:20.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:21.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:21.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:21.223 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:23:21.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:23:21.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:21.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:21.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:22.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:22.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:22.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:23:22.698 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:23:22.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:22.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:23.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:24.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:24.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:24.574 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:23:24.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:23:24.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:24.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:25.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:26.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:26.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:26.202 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:23:26.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:23:26.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:26.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:26.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:27.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:27.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:27.896 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:23:27.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:23:27.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:27.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:28.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:29.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:29.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:29.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:23:29.978 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:23:29.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:29.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:31.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:32.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:32.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:32.116 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:23:32.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:23:32.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:32.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:32.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:33.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:33.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:33.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:23:33.933 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:23:33.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:33.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:34.390 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1779122522,v1:127.0.0.1:6819/1779122522] [v2:127.0.0.1:6820/1779122522,v1:127.0.0.1:6821/1779122522] exists,up 58d268a7-eec2-4e03-8fee-7633b0a83ab6 2024-04-25T07:23:34.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:23:34.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:23:34.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:23:34.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-04-25T07:23:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:noout is set 2024-04-25T07:23:35.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-04-25T07:23:35.941 INFO:tasks.workunit.client.0.smithi102.stderr:noin is set 2024-04-25T07:23:35.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-04-25T07:23:36.753 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T07:23:36.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-04-25T07:23:36.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=test 2024-04-25T07:23:36.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-04-25T07:23:36.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=true 2024-04-25T07:23:36.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-04-25T07:23:36.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-04-25T07:23:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-04-25T07:23:37.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T07:23:38.537 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T07:23:38.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:23:39.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' true = true ']' 2024-04-25T07:23:39.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-04-25T07:23:40.670 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 allow_ec_overwrites to true 2024-04-25T07:23:40.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-04-25T07:23:40.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:23:40.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:23:40.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:23:40.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:23:40.695 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:23:40.695 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:23:40.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:23:40.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:23:40.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:23:40.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:23:40.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:23:40.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:23:40.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:23:40.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:23:40.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:23:41.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:23:41.261 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:23:41.261 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:23:41.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:23:41.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:41.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:23:41.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836504 2024-04-25T07:23:41.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836504 2024-04-25T07:23:41.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504' 2024-04-25T07:23:41.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:41.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:23:41.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672975 2024-04-25T07:23:41.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672975 2024-04-25T07:23:41.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672975' 2024-04-25T07:23:41.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:41.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:23:41.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509452 2024-04-25T07:23:41.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509452 2024-04-25T07:23:41.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672975 2-64424509452' 2024-04-25T07:23:41.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:41.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:41.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836504 2024-04-25T07:23:41.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:23:41.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836504 2024-04-25T07:23:41.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:41.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836504 2024-04-25T07:23:41.752 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836504 2024-04-25T07:23:41.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-04-25T07:23:41.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:23:42.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836504 2024-04-25T07:23:42.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:42.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672975 2024-04-25T07:23:42.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:42.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:23:42.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672975 2024-04-25T07:23:42.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:42.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672975 2024-04-25T07:23:42.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-04-25T07:23:42.151 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672975 2024-04-25T07:23:42.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:23:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672975 -lt 42949672975 2024-04-25T07:23:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:42.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509452 2024-04-25T07:23:42.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:42.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:23:42.594 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509452 2024-04-25T07:23:42.594 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:42.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509452 2024-04-25T07:23:42.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-04-25T07:23:42.595 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509452 2024-04-25T07:23:42.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:23:43.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509452 2024-04-25T07:23:43.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:23:44.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:23:44.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:23:44.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509452 -lt 64424509452 2024-04-25T07:23:44.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:23:44.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:23:44.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:23:44.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:23:44.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:23:44.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:23:44.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:23:44.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:23:44.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:23:44.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:23:44.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:23:45.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:23:45.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:23:45.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:23:45.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:23:45.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:23:45.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:23:45.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:23:45.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-04-25T07:23:45.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-04-25T07:23:45.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:23:45.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:23:46.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:23:46.379 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:23:46.380 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:23:46.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:23:46.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:46.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:23:46.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836507 2024-04-25T07:23:46.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836507 2024-04-25T07:23:46.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507' 2024-04-25T07:23:46.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:46.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:23:46.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-04-25T07:23:46.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-04-25T07:23:46.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672978' 2024-04-25T07:23:46.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:46.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:23:46.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-04-25T07:23:46.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-04-25T07:23:46.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672978 2-64424509455' 2024-04-25T07:23:46.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:46.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836507 2024-04-25T07:23:46.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:46.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:23:46.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836507 2024-04-25T07:23:46.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:46.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836507 2024-04-25T07:23:46.850 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836507 2024-04-25T07:23:46.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-04-25T07:23:46.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:23:47.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836507 2024-04-25T07:23:47.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:23:48.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:23:48.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:23:48.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836507 2024-04-25T07:23:48.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:48.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-04-25T07:23:48.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:48.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:23:48.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-04-25T07:23:48.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:48.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-04-25T07:23:48.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-25T07:23:48.713 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672978 2024-04-25T07:23:48.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:23:49.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672978 2024-04-25T07:23:49.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:49.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-04-25T07:23:49.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:49.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:23:49.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-04-25T07:23:49.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:49.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-04-25T07:23:49.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-04-25T07:23:49.158 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509455 2024-04-25T07:23:49.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:23:49.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509455 2024-04-25T07:23:49.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-04-25T07:23:49.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:23:49.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:23:49.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:23:49.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:23:49.576 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:23:49.576 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:23:49.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:23:49.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:23:49.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:23:49.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:23:49.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:23:49.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:23:49.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:23:49.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:23:49.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:23:50.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:23:50.105 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:23:50.105 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:23:50.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:23:50.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:50.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:23:50.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836510 2024-04-25T07:23:50.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836510 2024-04-25T07:23:50.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510' 2024-04-25T07:23:50.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:50.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:23:50.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672981 2024-04-25T07:23:50.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672981 2024-04-25T07:23:50.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672981' 2024-04-25T07:23:50.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:23:50.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:23:50.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509458 2024-04-25T07:23:50.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509458 2024-04-25T07:23:50.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672981 2-64424509458' 2024-04-25T07:23:50.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:50.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836510 2024-04-25T07:23:50.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:50.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:23:50.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836510 2024-04-25T07:23:50.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:50.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836510 2024-04-25T07:23:50.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-04-25T07:23:50.594 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836510 2024-04-25T07:23:50.594 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:23:50.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836510 2024-04-25T07:23:50.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:23:51.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:23:51.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:23:52.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836510 2024-04-25T07:23:52.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:52.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672981 2024-04-25T07:23:52.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:52.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:23:52.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672981 2024-04-25T07:23:52.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:52.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672981 2024-04-25T07:23:52.459 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672981 2024-04-25T07:23:52.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-04-25T07:23:52.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:23:52.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672981 -lt 42949672981 2024-04-25T07:23:52.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:23:52.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509458 2024-04-25T07:23:52.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:23:52.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:23:52.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-04-25T07:23:52.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:23:52.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509458 2024-04-25T07:23:52.890 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509458 2024-04-25T07:23:52.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-04-25T07:23:52.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:23:53.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509458 2024-04-25T07:23:53.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:23:53.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:23:53.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:23:53.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:23:53.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:23:53.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:23:53.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:23:53.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:23:53.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:23:53.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:23:53.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:23:54.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:23:54.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:23:54.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:23:54.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:23:54.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:23:54.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:23:54.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:23:54.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-04-25T07:23:54.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-04-25T07:23:55.220 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:23:55.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-04-25T07:23:55.235 INFO:tasks.workunit.client.0.smithi102.stdout:primary and soon to be divergent is 1 2024-04-25T07:23:55.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-04-25T07:23:55.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:23:55.613 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:23:55.613 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:23:40.032325+0000 0'0 28:17 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:39:31.492044+0000 0 0 2024-04-25T07:23:55.613 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:23:55.613 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:23:55.614 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:23:55.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-04-25T07:23:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:23:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-04-25T07:23:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-04-25T07:23:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:23:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-04-25T07:23:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-04-25T07:23:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:23:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-04-25T07:23:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-04-25T07:23:55.630 INFO:tasks.workunit.client.0.smithi102.stdout:writing initial objects 2024-04-25T07:23:55.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-04-25T07:23:55.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-04-25T07:23:55.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:55.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-04-25T07:23:55.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:55.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-04-25T07:23:55.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:55.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-04-25T07:23:56.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:56.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-04-25T07:23:56.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:56.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-04-25T07:23:56.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:56.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-04-25T07:23:57.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:57.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-04-25T07:23:57.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:57.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-04-25T07:23:57.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:57.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-04-25T07:23:57.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:57.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-04-25T07:23:57.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:57.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-04-25T07:23:58.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:58.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-04-25T07:23:58.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:58.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-04-25T07:23:58.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:58.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-04-25T07:23:58.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:58.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-04-25T07:23:59.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:59.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-04-25T07:23:59.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:59.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-04-25T07:23:59.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:59.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-04-25T07:23:59.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:23:59.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-04-25T07:24:00.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:00.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-04-25T07:24:00.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:00.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-04-25T07:24:00.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:00.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-04-25T07:24:00.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:00.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-04-25T07:24:01.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:01.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-04-25T07:24:01.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:01.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-04-25T07:24:01.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:01.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-04-25T07:24:01.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:01.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-04-25T07:24:02.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:02.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-04-25T07:24:02.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:02.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-04-25T07:24:02.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:02.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-04-25T07:24:02.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:02.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-04-25T07:24:03.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:03.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-04-25T07:24:03.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:03.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-04-25T07:24:03.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:03.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-04-25T07:24:03.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:03.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-04-25T07:24:04.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:04.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-04-25T07:24:04.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:04.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-04-25T07:24:04.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:04.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-04-25T07:24:04.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:04.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-04-25T07:24:05.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:05.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-04-25T07:24:05.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:05.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-04-25T07:24:05.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:05.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-04-25T07:24:05.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:05.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-04-25T07:24:06.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:06.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-04-25T07:24:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:06.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-04-25T07:24:06.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:06.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-04-25T07:24:06.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:06.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-04-25T07:24:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:07.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-04-25T07:24:07.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:07.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-04-25T07:24:07.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:07.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-04-25T07:24:08.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:08.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-04-25T07:24:08.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:08.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-04-25T07:24:08.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:08.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-04-25T07:24:08.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:08.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-04-25T07:24:08.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:08.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-04-25T07:24:09.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:09.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-04-25T07:24:09.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:09.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-04-25T07:24:09.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:09.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-04-25T07:24:10.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:10.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-04-25T07:24:10.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:10.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-04-25T07:24:10.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:10.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-04-25T07:24:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:10.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-04-25T07:24:11.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:11.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-04-25T07:24:11.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:11.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-04-25T07:24:11.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:11.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-04-25T07:24:11.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:11.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-04-25T07:24:12.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:12.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-04-25T07:24:12.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:12.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-04-25T07:24:12.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:12.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-04-25T07:24:12.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:12.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-04-25T07:24:13.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:13.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-04-25T07:24:13.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:13.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-04-25T07:24:13.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:13.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-04-25T07:24:13.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:13.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-04-25T07:24:14.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:14.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-04-25T07:24:14.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:14.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-04-25T07:24:14.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:14.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-04-25T07:24:14.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:14.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-04-25T07:24:15.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:15.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-04-25T07:24:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:15.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-04-25T07:24:15.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:15.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-04-25T07:24:15.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:15.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-04-25T07:24:15.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:15.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-04-25T07:24:16.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:16.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-04-25T07:24:16.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:16.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-04-25T07:24:16.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:16.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-04-25T07:24:17.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:17.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-04-25T07:24:17.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:17.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-04-25T07:24:17.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:17.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-04-25T07:24:17.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:17.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-04-25T07:24:18.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:18.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-04-25T07:24:18.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:18.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-04-25T07:24:18.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:18.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-04-25T07:24:18.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:18.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-04-25T07:24:19.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:19.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-04-25T07:24:19.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:19.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-04-25T07:24:19.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:19.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-04-25T07:24:19.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:19.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-04-25T07:24:20.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:20.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-04-25T07:24:20.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:20.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-04-25T07:24:20.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:24:20.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-04-25T07:24:20.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:24:20.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:24:20.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:24:20.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:24:20.604 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:24:20.604 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:24:20.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:24:20.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:24:20.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:24:20.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:24:20.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:24:20.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:24:20.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:24:20.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:24:20.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:24:21.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:24:21.121 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:24:21.122 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:24:21.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:24:21.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:21.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:24:21.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836518 2024-04-25T07:24:21.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836518 2024-04-25T07:24:21.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518' 2024-04-25T07:24:21.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:21.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:24:21.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672989 2024-04-25T07:24:21.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672989 2024-04-25T07:24:21.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518 1-42949672989' 2024-04-25T07:24:21.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:21.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:24:21.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509466 2024-04-25T07:24:21.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509466 2024-04-25T07:24:21.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836518 1-42949672989 2-64424509466' 2024-04-25T07:24:21.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:21.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836518 2024-04-25T07:24:21.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:21.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:24:21.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836518 2024-04-25T07:24:21.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:21.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836518 2024-04-25T07:24:21.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-04-25T07:24:21.610 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836518 2024-04-25T07:24:21.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:24:22.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836518 -lt 21474836518 2024-04-25T07:24:22.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:22.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672989 2024-04-25T07:24:22.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:22.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:24:22.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672989 2024-04-25T07:24:22.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:22.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672989 2024-04-25T07:24:22.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-04-25T07:24:22.036 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672989 2024-04-25T07:24:22.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:24:22.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672989 -lt 42949672989 2024-04-25T07:24:22.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:22.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509466 2024-04-25T07:24:22.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:22.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:24:22.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509466 2024-04-25T07:24:22.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:22.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509466 2024-04-25T07:24:22.450 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509466 2024-04-25T07:24:22.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-04-25T07:24:22.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:24:22.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509466 -lt 64424509466 2024-04-25T07:24:22.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:24:22.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:24:22.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:24:23.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:24:23.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:24:23.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:24:23.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:24:23.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:24:23.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:24:23.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:24:23.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:24:23.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:24:23.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:24:23.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:24:23.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:24:24.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:24:24.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:24:24.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:24:24.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-04-25T07:24:24.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:24:24.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:24:24.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:24:24.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:24:24.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-04-25T07:24:24.877 INFO:tasks.workunit.client.0.smithi102.stdout:blackholing osds 0 2 2024-04-25T07:24:24.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-04-25T07:24:24.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:24:25.284 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:24:25.284 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T07:23:40.032325+0000 28'100 28:117 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:39:31.492044+0000 0 0 2024-04-25T07:24:25.284 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:24:25.284 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:24:25.284 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:24:25.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:24:25.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-04-25T07:24:25.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:24:25.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:24:25.301 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:24:25.301 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:25.301 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:24:25.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.0.asok 2024-04-25T07:24:25.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:24:25.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:24:25.404 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:24:25.404 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:24:25.404 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:24:25.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:24:25.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-04-25T07:24:25.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:24:25.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:24:25.418 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:24:25.418 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:25.418 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:24:25.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.2.asok 2024-04-25T07:24:25.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:24:25.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:24:25.527 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:24:25.527 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:24:25.527 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:24:25.539 INFO:tasks.workunit.client.0.smithi102.stdout:writing divergent object 2024-04-25T07:24:25.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-04-25T07:24:25.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-04-25T07:24:25.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-04-25T07:24:26.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-04-25T07:24:26.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-04-25T07:24:26.764 INFO:tasks.workunit.client.0.smithi102.stdout:created pool test snap snap1 2024-04-25T07:24:26.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-04-25T07:24:26.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-04-25T07:24:26.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-04-25T07:24:36.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-04-25T07:24:36.774 INFO:tasks.workunit.client.0.smithi102.stderr:rados: no process found 2024-04-25T07:24:36.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-04-25T07:24:36.774 INFO:tasks.workunit.client.0.smithi102.stdout:killing all the osds 2024-04-25T07:24:36.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:24:37.181 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:24:37.182 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 active+clean 2024-04-25T07:23:40.032325+0000 29'104 29:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:37:51.918231+0000 0 0 2024-04-25T07:24:37.182 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:24:37.182 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:24:37.182 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:24:37.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:24:37.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:24:37.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:24:37.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:24:37.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:24:37.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:24:37.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:24:37.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:24:37.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-04-25T07:24:38.152 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T07:24:38.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:24:38.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-04-25T07:24:38.719 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already down. 2024-04-25T07:24:38.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:24:38.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-04-25T07:24:39.305 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already down. 2024-04-25T07:24:39.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:24:39.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-04-25T07:24:39.962 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. 2024-04-25T07:24:39.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:24:39.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-04-25T07:24:40.583 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already out. 2024-04-25T07:24:40.599 INFO:tasks.workunit.client.0.smithi102.stdout:bringing up non_divergent 0 2 2024-04-25T07:24:40.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-04-25T07:24:40.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:24:41.018 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:24:41.018 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 stale+active+clean 2024-04-25T07:23:40.032325+0000 29'104 29:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:37:51.918231+0000 0 0 2024-04-25T07:24:41.019 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:24:41.019 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:24:41.019 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:24:41.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:24:41.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:24:41.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:24:41.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:24:41.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:24:41.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:24:41.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:24:41.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:24:41.037 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:24:41.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:24:41.039 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:24:41.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:24:41.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:24:41.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:24:41.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:24:41.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:24:41.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:24:41.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:24:41.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:24:41.081+0000 7f0214a917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:41.079 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:24:41.081+0000 7f0214a917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:41.081 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:24:41.085+0000 7f0214a917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:41.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:24:41.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:24:41.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:24:41.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:24:41.504 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:24:41.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:24:41.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:41.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:24:41.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:41.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:41.647 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:24:41.649+0000 7f0214a917c0 -1 Falling back to public interface 2024-04-25T07:24:41.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:42.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:42.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:42.972 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:24:42.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:24:42.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:42.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:43.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:44.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:44.407 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:24:44.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:44.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:24:44.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:44.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:44.750 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:24:44.753+0000 7f0214a917c0 -1 osd.0 28 log_to_monitors true 2024-04-25T07:24:44.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:45.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:45.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:45.834 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:24:45.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:24:45.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:45.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:24:46.276 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/3630948605,v1:127.0.0.1:6803/3630948605] [v2:127.0.0.1:6804/3630948605,v1:127.0.0.1:6805/3630948605] exists,up 87e4e2ff-6f89-4a5e-9a0b-7ee1f6361e7e 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:24:46.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:24:46.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:24:46.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:24:46.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:24:46.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:24:46.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:24:46.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:24:46.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:24:46.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:24:46.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:24:46.282 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:24:46.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:24:46.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:24:46.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:24:46.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:24:46.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:24:46.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:24:46.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:24:46.317 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:24:46.317+0000 7f461a9227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:46.328 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:24:46.329+0000 7f461a9227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:46.329 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:24:46.329+0000 7f461a9227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:24:46.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:24:46.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:24:46.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:24:46.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:24:46.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:24:46.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:46.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:24:46.713 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:24:46.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:46.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:46.875 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:24:46.877+0000 7f461a9227c0 -1 Falling back to public interface 2024-04-25T07:24:47.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:48.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:48.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:48.126 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:24:48.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:24:48.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:48.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:48.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:49.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:49.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:49.543 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:24:49.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:24:49.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:49.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:49.963 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:24:49.965+0000 7f461a9227c0 -1 osd.2 28 log_to_monitors true 2024-04-25T07:24:49.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:50.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:50.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:50.989 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:24:50.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:24:50.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:50.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:51.461 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/335452904,v1:127.0.0.1:6811/335452904] [v2:127.0.0.1:6812/335452904,v1:127.0.0.1:6813/335452904] exists,up 58d268a7-eec2-4e03-8fee-7633b0a83ab6 2024-04-25T07:24:51.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:24:51.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:24:51.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:24:51.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:24:51.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-04-25T07:24:52.126 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already in. 2024-04-25T07:24:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:24:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-04-25T07:24:53.035 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already in. 2024-04-25T07:24:53.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-04-25T07:24:58.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-04-25T07:24:58.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-04-25T07:24:58.065 INFO:tasks.workunit.client.0.smithi102.stdout:writing non-divergent object existing_4 2024-04-25T07:24:58.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-04-25T07:24:58.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:24:58.498 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:24:58.498 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:24:53.491398+0000 28'100 43:128 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:03:28.041473+0000 0 0 2024-04-25T07:24:58.498 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:24:58.498 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:24:58.498 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:24:58.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:24:58.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:24:58.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-04-25T07:24:58.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:24:58.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:24:58.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:24:58.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:24:58.635 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:24:58.635 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:24:58.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:24:58.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:24:58.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:24:58.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:24:58.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:24:58.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:24:58.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:24:58.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:24:58.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:24:59.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:24:59.131 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:24:59.131 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:24:59.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:24:59.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:59.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:24:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822662 2024-04-25T07:24:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822662 2024-04-25T07:24:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822662' 2024-04-25T07:24:59.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:59.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:24:59.414 INFO:tasks.workunit.client.0.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:24:59.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:24:59.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:24:59.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:24:59.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:59.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:24:59.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757252 2024-04-25T07:24:59.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757252 2024-04-25T07:24:59.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822662 2-163208757252' 2024-04-25T07:24:59.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:59.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-154618822662 2024-04-25T07:24:59.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:59.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:24:59.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822662 2024-04-25T07:24:59.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:59.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822662 2024-04-25T07:24:59.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 154618822662' 2024-04-25T07:24:59.567 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 154618822662 2024-04-25T07:24:59.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:24:59.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822662 -lt 154618822662 2024-04-25T07:24:59.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:59.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-163208757252 2024-04-25T07:24:59.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:59.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:24:59.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757252 2024-04-25T07:24:59.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:59.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757252 2024-04-25T07:24:59.981 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 163208757252 2024-04-25T07:24:59.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 163208757252' 2024-04-25T07:24:59.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:25:00.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757252 -lt 163208757252 2024-04-25T07:25:00.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:25:00.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:00.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:00.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:25:00.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:00.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:00.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:00.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:00.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:00.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:00.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:01.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:01.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:01.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:01.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:25:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:25:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:25:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:25:02.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:02.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:02.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:02.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:02.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:02.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:02.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:02.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:02.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:02.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:02.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:02.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:03.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:03.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:03.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:03.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:03.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:03.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:03.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:03.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:03.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:03.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:03.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:03.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:25:03.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:03.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:25:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:03.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:03.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:03.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:03.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:03.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:03.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:03.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:04.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:04.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:04.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:04.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:04.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:04.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:05.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:05.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:05.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:25:05.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:05.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:25:05.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:05.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:05.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:05.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:05.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:05.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:05.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:05.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:06.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:06.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:06.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:06.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:06.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:06.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:06.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:06.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:06.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:06.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:06.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:06.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:06.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:07.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:07.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:07.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:25:07.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:07.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:25:08.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:08.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:08.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:08.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:08.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:08.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:08.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:08.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:08.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:08.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:08.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:08.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:08.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:08.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:08.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:08.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:08.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:08.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:08.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:08.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:08.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:09.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:09.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:09.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:25:09.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:09.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:25:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:11.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:11.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:11.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:11.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:11.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:11.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:11.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:11.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:11.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:11.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:12.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:12.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:12.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:12.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:12.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:12.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:12.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:12.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:12.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:25:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:25:15.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:15.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:15.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:15.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:15.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:15.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:15.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:15.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:16.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:16.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:16.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:16.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:16.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:16.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:16.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:16.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:16.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:16.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:16.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:16.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:16.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:17.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:17.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:17.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:25:17.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:17.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:25:23.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:23.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:23.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:23.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:23.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:23.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:23.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:23.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:24.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:24.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:24.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:24.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:24.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:24.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:24.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:24.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:24.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:24.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:24.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:24.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:24.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:25.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:25.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:25.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:25:25.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:25.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:25:32.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:32.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:32.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:32.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:32.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:32.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:32.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:32.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:32.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:32.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:32.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:32.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:33.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:33.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:33.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:33.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:33.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:33.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:33.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:33.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:33.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:34.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:34.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:34.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:25:34.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:25:34.629 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:25:34.629 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_fingerprint": "74422c0f-4f3e-48ec-8b72-c3b5e4a37906", 2024-04-25T07:25:34.629 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "timestamp": "2024-04-25T07:25:34.634465+0000", 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "tag": "", 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "health": { 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "status": "HEALTH_WARN", 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "checks": { 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 17 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [], 2024-04-25T07:25:34.630 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: "OSD_DOWN": { 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 osds down", 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "osd.1 (root=default,host=smithi102) is down" 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.631 INFO:tasks.workunit.client.0.smithi102.stdout: "PG_DEGRADED": { 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:25:34.632 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: "mutes": [] 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.633 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_first_committed": 1, 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_last_committed": 1, 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap": { 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:21:41.087851Z", 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:21:41.087851Z", 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release": 19, 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release_name": "squid", 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "election_strategy": 1, 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "disallowed_leaders: ": "", 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": false, 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "tiebreaker_mon": "", 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_ranks: ": "", 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "features": { 2024-04-25T07:25:34.634 INFO:tasks.workunit.client.0.smithi102.stdout: "persistent": [ 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "kraken", 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "luminous", 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "mimic", 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap-prune", 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "nautilus", 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "octopus", 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "pacific", 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "elector-pinging", 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "quincy", 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "reef", 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "squid" 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "optional": [] 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.635 INFO:tasks.workunit.client.0.smithi102.stdout: "mons": [ 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: "rank": 0, 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "a", 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 0 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:25:34.636 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 0, 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 0, 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_location": "{}" 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: "quorum": [ 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: 0 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap": { 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 43, 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:21:42.221989+0000", 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:24:53.113508+0000", 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: "last_up_change": "2024-04-25T07:24:50.356218+0000", 2024-04-25T07:25:34.637 INFO:tasks.workunit.client.0.smithi102.stdout: "last_in_change": "2024-04-25T07:24:52.552403+0000", 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_num": 5800384, 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_set": [ 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "nodown", 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "noin", 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "noout", 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "pglog_hardlimit", 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snapdirs", 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "recovery_deletes", 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "sortbitwise" 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_version": 10, 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:25:34.638 INFO:tasks.workunit.client.0.smithi102.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_snapshot": "", 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_max": 1, 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "max_osd": 3, 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "min_compat_client": "jewel", 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "require_osd_release": "squid", 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "allow_crimson": false, 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "pools": [ 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_name": "test", 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "create_time": "2024-04-25T07:23:38.190372+0000", 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": 16389, 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-04-25T07:25:34.639 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 3, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 3, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "min_size": 2, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_rule": 1, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "object_hash": 2, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num": 1, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num": 1, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num_target": 1, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_target": 1, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_pending": 1, 2024-04-25T07:25:34.640 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pg_merge_meta": { 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "source_pgid": "0.0", 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "ready_epoch": 0, 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_started": 0, 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": 0, 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "source_version": "0'0", 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "target_version": "0'0" 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "last_change": "29", 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend": "0", 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "auid": 0, 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_mode": "pool", 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_seq": 1, 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_epoch": 29, 2024-04-25T07:25:34.641 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_snaps": [ 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 1, 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-04-25T07:24:26.590613+0000", 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap1" 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps": "[]", 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_bytes": 0, 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_objects": 0, 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "tiers": [], 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "tier_of": -1, 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "read_tier": -1, 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "write_tier": -1, 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_mode": "none", 2024-04-25T07:25:34.642 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_bytes": 0, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_objects": 0, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_flush_age": 0, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_evict_age": 0, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profile": "myprofile", 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_params": { 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "none" 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_period": 0, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_count": 0, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "use_gmt_hitset": true, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:25:34.643 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "grade_table": [], 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "stripe_width": 8192, 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_num_objects": 0, 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "fast_read": false, 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "options": {}, 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "application_metadata": {} 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "osds": [ 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "87e4e2ff-6f89-4a5e-9a0b-7ee1f6361e7e", 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:25:34.644 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 5, 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 28, 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 36, 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 41, 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 30, 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:25:34.645 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3630948605 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3630948605 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.646 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3630948605 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3630948605 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3630948605 2024-04-25T07:25:34.647 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3630948605 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3630948605 2024-04-25T07:25:34.648 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3630948605 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6803/3630948605", 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6805/3630948605", 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3630948605", 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3630948605", 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:25:34.649 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "7e6be8e7-02ca-4c7c-a7ae-17fd198b434f", 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 0, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 0, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 0, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 10, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 24, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 31, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:25:34.650 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.651 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.652 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.653 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/2403496704", 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/2403496704", 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2403496704", 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2403496704", 2024-04-25T07:25:34.654 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "exists" 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "58d268a7-eec2-4e03-8fee-7633b0a83ab6", 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 15, 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 28, 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 38, 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 0, 2024-04-25T07:25:34.655 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 32, 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 335452904 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 335452904 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.656 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 335452904 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 335452904 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:25:34.657 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 335452904 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 335452904 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:25:34.658 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 335452904 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 335452904 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/335452904", 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/335452904", 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/335452904", 2024-04-25T07:25:34.659 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/335452904", 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_xinfo": [ 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:24:37.912303+0000", 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:22:17.852863+0000", 2024-04-25T07:25:34.660 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:24:38.554051+0000", 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:22:56.014950+0000", 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:24:39.134687+0000", 2024-04-25T07:25:34.661 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:23:33.312812+0000", 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap": [], 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_items": [], 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_primaries": [], 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_temp": [], 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_temp": [], 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "blocklist": {}, 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "range_blocklist": {}, 2024-04-25T07:25:34.662 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profiles": { 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "default": { 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "k": "2", 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "m": "2", 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "plugin": "jerasure", 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "technique": "reed_sol_van" 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "myprofile": { 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "crush-device-class": "", 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "crush-failure-domain": "osd", 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "crush-num-failure-domains": "0", 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "crush-osds-per-failure-domain": "0", 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "crush-root": "default", 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-25T07:25:34.663 INFO:tasks.workunit.client.0.smithi102.stdout: "k": "2", 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "m": "1", 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "plugin": "jerasure", 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "technique": "reed_sol_van", 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "w": "8" 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps_queue": [], 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "new_removed_snaps": [], 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "new_purged_snaps": [], 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_node_flags": {}, 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "device_class_flags": {}, 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": { 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_enabled": false, 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_bucket_count": 0, 2024-04-25T07:25:34.664 INFO:tasks.workunit.client.0.smithi102.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_bucket": 0 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_metadata": [ 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6804/3630948605,v1:127.0.0.1:6805/3630948605]", 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:25:34.665 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:25:34.666 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:21:49.180449Z", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6802/3630948605,v1:127.0.0.1:6803/3630948605]", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3630948605,v1:127.0.0.1:6809/3630948605]", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3630948605,v1:127.0.0.1:6807/3630948605]", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:25:34.667 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/2403496704,v1:127.0.0.1:6813/2403496704]", 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:25:34.668 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:25:34.669 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:22:26.685194Z", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:25:34.670 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/2403496704,v1:127.0.0.1:6811/2403496704]", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2403496704,v1:127.0.0.1:6817/2403496704]", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2403496704,v1:127.0.0.1:6815/2403496704]", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:25:34.671 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/335452904,v1:127.0.0.1:6813/335452904]", 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:25:34.672 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:25:34.673 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:23:04.420586Z", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/335452904,v1:127.0.0.1:6811/335452904]", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/335452904,v1:127.0.0.1:6817/335452904]", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/335452904,v1:127.0.0.1:6815/335452904]", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:25:34.674 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_clean_epochs": { 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": { 2024-04-25T07:25:34.675 INFO:tasks.workunit.client.0.smithi102.stdout: "per_pool": [] 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_epochs": [ 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 36 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 38 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_first_committed": 1, 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_last_committed": 43, 2024-04-25T07:25:34.676 INFO:tasks.workunit.client.0.smithi102.stdout: "crushmap": { 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": [ 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.0", 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.1", 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.2", 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:25:34.677 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: "types": [ 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 0, 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd" 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "host" 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 2, 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "chassis" 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.678 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 3, 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "rack" 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 4, 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "row" 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 5, 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pdu" 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 6, 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pod" 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.679 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 7, 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "room" 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 8, 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "datacenter" 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 9, 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "zone" 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 10, 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "region" 2024-04-25T07:25:34.680 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "root" 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "buckets": [ 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -1, 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default", 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:25:34.681 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -2, 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default~hdd", 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.682 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102", 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:25:34.683 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:25:34.684 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102~hdd", 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.685 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: "rules": [ 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_id": 0, 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_name": "replicated_rule", 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: "steps": [ 2024-04-25T07:25:34.686 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "take", 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: "item": -1, 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: "item_name": "default" 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "choose_firstn", 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 0, 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "osd" 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "emit" 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.687 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_id": 1, 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_name": "test", 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 3, 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: "steps": [ 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "set_chooseleaf_tries", 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 5 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "set_choose_tries", 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 100 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "take", 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: "item": -1, 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: "item_name": "default" 2024-04-25T07:25:34.688 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "choose_indep", 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 0, 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "osd" 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "emit" 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.689 INFO:tasks.workunit.client.0.smithi102.stdout: "tunables": { 2024-04-25T07:25:34.690 INFO:tasks.workunit.client.0.smithi102.stderr:report 4111344588 2024-04-25T07:25:34.690 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_tries": 0, 2024-04-25T07:25:34.690 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:25:34.690 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_total_tries": 50, 2024-04-25T07:25:34.690 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:25:34.690 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:25:34.690 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_stable": 1, 2024-04-25T07:25:34.690 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_descents": 100, 2024-04-25T07:25:34.690 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_collision_tries": 100, 2024-04-25T07:25:34.690 INFO:tasks.workunit.client.0.smithi102.stdout: "straw_calc_version": 1, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "profile": "jewel", 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_tunables": 1, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "legacy_tunables": 0, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "minimum_required_version": "jewel", 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables": 1, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables2": 1, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v2_rules": 1, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables3": 1, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v3_rules": 0, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v4_buckets": 1, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables5": 1, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v5_rules": 0, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "has_msr_rules": 0 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.691 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_args": {} 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: "fsmap": { 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: "default_fscid": -1, 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": { 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": {}, 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: "ro_compat": {}, 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: "incompat": { 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_1": "base v0.20", 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:25:34.692 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_8": "no anchor table", 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_9": "file layout v2", 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_11": "minor log segments", 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_flags": { 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "enable_multiple": true, 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "ever_enabled_multiple": true 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "standbys": [], 2024-04-25T07:25:34.693 INFO:tasks.workunit.client.0.smithi102.stdout: "filesystems": [] 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "auth": { 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 5, 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "num_secrets": 3 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 1, 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_active": 1, 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_unknown": 0, 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osd": 3, 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_sum": { 2024-04-25T07:25:34.694 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 62667, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 101, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 1, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 303, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 101, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 101, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 101, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 101, 2024-04-25T07:25:34.695 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:25:34.696 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:25:34.697 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 0, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 0, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 18, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 18, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 3, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 3, 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 0 2024-04-25T07:25:34.698 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum": { 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 3, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83660, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2156, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314489140, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:25:34.699 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036879360, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2207744, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 1549883, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10396, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416932 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:25:34.700 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum_by_class": { 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: "hdd": { 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:25:34.701 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 3, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 83660, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2156, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 10, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 81461, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314489140, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322036879360, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2207744, 2024-04-25T07:25:34.702 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 1549883, 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 10396, 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 83416932 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:25:34.703 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:25:34.704 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:25:34.704 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:25:34.704 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:25:34.704 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.704 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:25:34.705 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:25:34.705 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.705 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.705 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_stats": [ 2024-04-25T07:25:34.705 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.705 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T07:25:34.705 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 1, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 62667, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 101, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 1, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 303, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 101, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 101, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:25:34.706 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 101, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 101, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:25:34.707 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:25:34.708 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 1241088, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 1241088, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 18, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 18, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 3, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 3, 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 3 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.709 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_stats": [ 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 154618822669 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 42949672993 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 163208757259 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.710 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_state": [ 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "active+undersized+degraded", 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 1 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_osd": [ 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2147483647, 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:25:34.711 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 1, 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 0, 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:25:34.712 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": [ 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": {} 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "servicemap": { 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 5, 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:24:35.641582+0000", 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "services": {} 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_last_committed": 105, 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_first_committed": 1, 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_last_committed": 406, 2024-04-25T07:25:34.713 INFO:tasks.workunit.client.0.smithi102.stdout: "paxos": { 2024-04-25T07:25:34.714 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:25:34.714 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 561, 2024-04-25T07:25:34.714 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pn": 0, 2024-04-25T07:25:34.714 INFO:tasks.workunit.client.0.smithi102.stdout: "accepted_pn": 0 2024-04-25T07:25:34.714 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:25:34.714 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:25:34.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:25:34.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:25:34.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-04-25T07:25:34.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:25:34.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:25:34.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:25:34.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:25:34.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:25:34.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:25:34.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-04-25T07:25:34.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:25:34.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:25:34.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-25T07:25:34.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:25:34.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-04-25T07:25:34.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-04-25T07:25:36.111 INFO:tasks.workunit.client.0.smithi102.stderr:PG '1.0' not found 2024-04-25T07:25:36.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:25:36.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-04-25T07:25:36.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:25:36.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:25:36.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:25:36.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:25:36.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:25:36.411 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:25:36.411 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:36.411 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:25:36.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:25:36.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:25:36.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:25:36.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:25:36.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:25:36.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:25:36.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:25:36.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:25:36.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:25:36.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:25:36.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:25:36.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:25:36.413 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:25:36.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:25:36.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:25:36.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:25:36.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:25:36.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:25:36.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:25:36.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:25:36.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:25:36.456+0000 7f7b505107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:36.457 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:25:36.460+0000 7f7b505107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:36.459 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:25:36.460+0000 7f7b505107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:36.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:25:36.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:25:36.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:25:36.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:25:36.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:25:36.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:36.835 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:25:36.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:25:36.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:36.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:25:37.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:25:37.020+0000 7f7b505107c0 -1 Falling back to public interface 2024-04-25T07:25:37.282 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 36 up_thru 41 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/3630948605,v1:127.0.0.1:6803/3630948605] [v2:127.0.0.1:6804/3630948605,v1:127.0.0.1:6805/3630948605] exists,up 87e4e2ff-6f89-4a5e-9a0b-7ee1f6361e7e 2024-04-25T07:25:37.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:25:37.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:25:37.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:25:37.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:25:37.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-04-25T07:25:37.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:25:37.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:25:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:25:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:25:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:25:37.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:25:37.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-04-25T07:25:37.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:25:37.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:25:37.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-25T07:25:37.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:25:37.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-04-25T07:25:37.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-04-25T07:25:38.730 INFO:tasks.workunit.client.0.smithi102.stderr:PG '1.0' not found 2024-04-25T07:25:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:25:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-04-25T07:25:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:25:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:25:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:25:39.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:25:39.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:25:39.023 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:25:39.023 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:39.023 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:25:39.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:25:39.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:25:39.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:25:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:25:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:25:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:25:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:25:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:25:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:25:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:25:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:25:39.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:25:39.024 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:25:39.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:25:39.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:25:39.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:25:39.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:25:39.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:25:39.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:25:39.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:25:39.074 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:25:39.076+0000 7f04d3f387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:39.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:25:39.076+0000 7f04d3f387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:39.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:25:39.080+0000 7f04d3f387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:39.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:25:39.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:25:39.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:25:39.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:25:39.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:25:39.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:39.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:25:39.460 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:25:39.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:39.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:39.634 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:25:39.636+0000 7f04d3f387c0 -1 Falling back to public interface 2024-04-25T07:25:39.660 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:25:39.660+0000 7f7b505107c0 -1 osd.0 43 log_to_monitors true 2024-04-25T07:25:39.973 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/335452904,v1:127.0.0.1:6811/335452904] [v2:127.0.0.1:6812/335452904,v1:127.0.0.1:6813/335452904] exists,up 58d268a7-eec2-4e03-8fee-7633b0a83ab6 2024-04-25T07:25:39.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:25:39.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:25:39.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:25:39.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-04-25T07:25:39.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:25:39.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:25:39.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:25:39.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:25:39.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:25:39.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-04-25T07:25:41.869 INFO:tasks.workunit.client.0.smithi102.stderr:PG '1.0' not found 2024-04-25T07:25:42.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:25:42.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:25:42.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-04-25T07:25:42.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:25:42.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:25:42.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:25:42.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:25:42.160 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:25:42.160 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:25:42.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:25:42.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:25:42.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:25:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:25:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:25:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:25:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:25:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:25:42.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:25:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:25:42.400+0000 7f04d3f387c0 -1 osd.2 43 log_to_monitors true 2024-04-25T07:25:42.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:25:42.748 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:25:42.748 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:25:42.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:25:42.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:25:42.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:25:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=197568495619 2024-04-25T07:25:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 197568495619 2024-04-25T07:25:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-197568495619' 2024-04-25T07:25:42.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:25:42.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:25:43.111 INFO:tasks.workunit.client.0.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:25:43.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:25:43.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:25:43.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:25:43.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:25:43.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:25:43.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364803 2024-04-25T07:25:43.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364803 2024-04-25T07:25:43.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-197568495619 2-214748364803' 2024-04-25T07:25:43.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:25:43.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-197568495619 2024-04-25T07:25:43.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:25:43.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:25:43.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-197568495619 2024-04-25T07:25:43.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:25:43.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=197568495619 2024-04-25T07:25:43.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2024-04-25T07:25:43.340 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 197568495619 2024-04-25T07:25:43.340 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:25:43.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 197568495619 2024-04-25T07:25:43.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:25:44.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:25:44.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:25:45.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 197568495619 -lt 197568495619 2024-04-25T07:25:45.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:25:45.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-214748364803 2024-04-25T07:25:45.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:25:45.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:25:45.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:25:45.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-214748364803 2024-04-25T07:25:45.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364803 2024-04-25T07:25:45.233 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 214748364803 2024-04-25T07:25:45.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-04-25T07:25:45.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:25:45.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364803 -lt 214748364803 2024-04-25T07:25:45.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:25:45.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:45.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:46.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:25:46.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:46.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:46.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:46.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:46.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:46.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:46.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:46.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:46.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:46.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:46.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:47.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:47.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:25:47.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:25:47.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:25:47.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:47.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:25:47.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:47.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:47.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:47.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:47.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:47.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:47.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:47.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:47.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:47.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:47.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:47.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:48.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:48.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:48.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:48.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:48.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:48.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:48.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:48.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:48.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:48.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:48.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:48.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:25:48.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:48.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:25:49.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:49.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:49.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:49.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:49.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:49.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:49.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:49.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:49.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:49.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:49.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:49.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:50.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:50.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:50.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:50.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:50.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:50.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:50.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:50.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:50.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:50.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:50.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:50.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:25:50.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:50.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:25:50.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:50.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:50.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:50.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:50.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:50.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:50.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:50.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:51.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:51.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:51.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:51.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:51.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:51.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:51.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:51.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:51.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:51.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:51.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:51.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:51.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:52.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:52.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:52.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:25:52.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:52.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:25:53.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:53.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:53.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:53.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:53.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:53.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:53.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:53.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:53.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:53.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:53.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:53.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:54.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:54.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:54.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:54.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:54.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:54.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:54.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:54.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:54.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:54.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:54.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:54.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:25:54.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:54.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:25:56.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:56.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:56.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:56.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:56.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:56.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:56.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:56.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:56.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:56.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:56.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:56.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:57.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:57.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:57.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:57.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:57.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:57.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:57.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:57.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:57.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:57.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:57.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:57.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:25:57.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:57.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:26:01.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:01.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:01.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:01.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:01.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:01.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:01.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:01.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:01.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:01.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:01.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:01.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:02.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:02.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:02.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:02.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:02.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:02.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:02.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:02.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:02.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:02.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:02.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:02.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:26:02.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:26:02.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:26:09.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:09.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:09.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:09.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:09.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:09.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:09.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:09.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:09.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:09.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:09.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:09.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:09.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:09.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:09.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:09.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:09.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:09.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:09.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:09.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:09.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:10.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:10.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:10.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:26:10.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:26:10.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:26:17.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:17.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:17.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:17.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:17.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:17.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:17.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:17.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:18.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:18.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:18.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:18.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:18.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:18.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:18.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:18.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:18.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:18.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:18.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:18.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:18.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:19.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:19.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:19.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:26:19.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:26:19.962 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:26:19.962 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_fingerprint": "74422c0f-4f3e-48ec-8b72-c3b5e4a37906", 2024-04-25T07:26:19.962 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "timestamp": "2024-04-25T07:26:19.967166+0000", 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "tag": "", 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "health": { 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "status": "HEALTH_WARN", 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "checks": { 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 17 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [], 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:26:19.963 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: "OSD_DOWN": { 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 osds down", 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "osd.1 (root=default,host=smithi102) is down" 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: "PG_DEGRADED": { 2024-04-25T07:26:19.964 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:19.965 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.966 INFO:tasks.workunit.client.0.smithi102.stdout: "mutes": [] 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_first_committed": 1, 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_last_committed": 1, 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap": { 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:21:41.087851Z", 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:21:41.087851Z", 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release": 19, 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release_name": "squid", 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "election_strategy": 1, 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "disallowed_leaders: ": "", 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": false, 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "tiebreaker_mon": "", 2024-04-25T07:26:19.967 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_ranks: ": "", 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "features": { 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "persistent": [ 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "kraken", 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "luminous", 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "mimic", 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap-prune", 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "nautilus", 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "octopus", 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "pacific", 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "elector-pinging", 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "quincy", 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "reef", 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "squid" 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:19.968 INFO:tasks.workunit.client.0.smithi102.stdout: "optional": [] 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: "mons": [ 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: "rank": 0, 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "a", 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 0 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:26:19.969 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 0, 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 0, 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_location": "{}" 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: "quorum": [ 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: 0 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap": { 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 52, 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "5aac55d2-2692-43c2-a04d-c7f10efd07d7", 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T07:21:42.221989+0000", 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:25:43.591177+0000", 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: "last_up_change": "2024-04-25T07:25:42.870583+0000", 2024-04-25T07:26:19.970 INFO:tasks.workunit.client.0.smithi102.stdout: "last_in_change": "2024-04-25T07:24:52.552403+0000", 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_num": 5800384, 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_set": [ 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "nodown", 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "noin", 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "noout", 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "pglog_hardlimit", 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snapdirs", 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "recovery_deletes", 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "sortbitwise" 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_version": 12, 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:26:19.971 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_snapshot": "", 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_max": 1, 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "max_osd": 3, 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "min_compat_client": "jewel", 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "require_osd_release": "squid", 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "allow_crimson": false, 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "pools": [ 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_name": "test", 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "create_time": "2024-04-25T07:23:38.190372+0000", 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": 16389, 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 3, 2024-04-25T07:26:19.972 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 3, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "min_size": 2, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_rule": 1, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "object_hash": 2, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num": 1, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num": 1, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num_target": 1, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_target": 1, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_pending": 1, 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pg_merge_meta": { 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "source_pgid": "0.0", 2024-04-25T07:26:19.973 INFO:tasks.workunit.client.0.smithi102.stdout: "ready_epoch": 0, 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_started": 0, 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": 0, 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "source_version": "0'0", 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "target_version": "0'0" 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "last_change": "29", 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend": "0", 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "auid": 0, 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_mode": "pool", 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_seq": 1, 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_epoch": 29, 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_snaps": [ 2024-04-25T07:26:19.974 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "snapid": 1, 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "stamp": "2024-04-25T07:24:26.590613+0000", 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "snap1" 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps": "[]", 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_bytes": 0, 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_objects": 0, 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "tiers": [], 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "tier_of": -1, 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "read_tier": -1, 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "write_tier": -1, 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_mode": "none", 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_bytes": 0, 2024-04-25T07:26:19.975 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_objects": 0, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_flush_age": 0, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_evict_age": 0, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profile": "myprofile", 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_params": { 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "none" 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_period": 0, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_count": 0, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "use_gmt_hitset": true, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:26:19.976 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "grade_table": [], 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "stripe_width": 8192, 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_num_objects": 0, 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "fast_read": false, 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "options": {}, 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "application_metadata": {} 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "osds": [ 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "87e4e2ff-6f89-4a5e-9a0b-7ee1f6361e7e", 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:26:19.977 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 36, 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 43, 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 46, 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 51, 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 45, 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2563715480 2024-04-25T07:26:19.978 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2563715480 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2563715480 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.979 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2563715480 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2563715480 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.980 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2563715480 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2563715480 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:26:19.981 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2563715480 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6803/2563715480", 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6805/2563715480", 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2563715480", 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2563715480", 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:26:19.982 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "7e6be8e7-02ca-4c7c-a7ae-17fd198b434f", 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 0, 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 0, 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 0, 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 10, 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 24, 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 31, 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:26:19.983 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.984 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.985 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:26:19.986 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2403496704 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/2403496704", 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/2403496704", 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2403496704", 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2403496704", 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: "exists" 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "58d268a7-eec2-4e03-8fee-7633b0a83ab6", 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T07:26:19.987 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 38, 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 43, 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 50, 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 0, 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 49, 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3198035605 2024-04-25T07:26:19.988 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3198035605 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3198035605 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.989 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3198035605 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3198035605 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.990 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3198035605 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3198035605 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T07:26:19.991 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 3198035605 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/3198035605", 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/3198035605", 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3198035605", 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3198035605", 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:19.992 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_xinfo": [ 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:25:40.143883+0000", 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:22:17.852863+0000", 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:26:19.993 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:24:38.554051+0000", 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:22:56.014950+0000", 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "2024-04-25T07:25:42.653039+0000", 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T07:26:19.994 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T07:23:33.312812+0000", 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap": [], 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_items": [], 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_primaries": [], 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_temp": [], 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_temp": [], 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "blocklist": {}, 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "range_blocklist": {}, 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profiles": { 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "default": { 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "k": "2", 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "m": "2", 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "plugin": "jerasure", 2024-04-25T07:26:19.995 INFO:tasks.workunit.client.0.smithi102.stdout: "technique": "reed_sol_van" 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "myprofile": { 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "crush-device-class": "", 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "crush-failure-domain": "osd", 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "crush-num-failure-domains": "0", 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "crush-osds-per-failure-domain": "0", 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "crush-root": "default", 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "k": "2", 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "m": "1", 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "plugin": "jerasure", 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "technique": "reed_sol_van", 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: "w": "8" 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.996 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps_queue": [], 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "new_removed_snaps": [], 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "new_purged_snaps": [], 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_node_flags": {}, 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "device_class_flags": {}, 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": { 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_enabled": false, 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_bucket_count": 0, 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_bucket": 0 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_metadata": [ 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:19.997 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6804/2563715480,v1:127.0.0.1:6805/2563715480]", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:26:19.998 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:21:49.180449Z", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:26:19.999 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6802/2563715480,v1:127.0.0.1:6803/2563715480]", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2563715480,v1:127.0.0.1:6809/2563715480]", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2563715480,v1:127.0.0.1:6807/2563715480]", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:26:20.000 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/2403496704,v1:127.0.0.1:6813/2403496704]", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:26:20.001 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:20.002 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:22:26.685194Z", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/2403496704,v1:127.0.0.1:6811/2403496704]", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2403496704,v1:127.0.0.1:6817/2403496704]", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2403496704,v1:127.0.0.1:6815/2403496704]", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:26:20.003 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T07:26:20.004 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/3198035605,v1:127.0.0.1:6813/3198035605]", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:26:20.005 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T07:23:04.420586Z", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T07:26:20.006 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/3198035605,v1:127.0.0.1:6811/3198035605]", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3198035605,v1:127.0.0.1:6817/3198035605]", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3198035605,v1:127.0.0.1:6815/3198035605]", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T07:26:20.007 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_clean_epochs": { 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: "min_last_epoch_clean": 25, 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": { 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: "per_pool": [ 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: "floor": 25 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_epochs": [ 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:26:20.008 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 46 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 50 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_first_committed": 1, 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_last_committed": 52, 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: "crushmap": { 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": [ 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.0", 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:26:20.009 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.1", 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.2", 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: "types": [ 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.010 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 0, 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd" 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "host" 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 2, 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "chassis" 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 3, 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "rack" 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.011 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 4, 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "row" 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 5, 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pdu" 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 6, 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pod" 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 7, 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "room" 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.012 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 8, 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "datacenter" 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 9, 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "zone" 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 10, 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "region" 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "root" 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:20.013 INFO:tasks.workunit.client.0.smithi102.stdout: "buckets": [ 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -1, 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default", 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:26:20.014 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -2, 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default~hdd", 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:26:20.015 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102", 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:26:20.016 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.017 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102~hdd", 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 19206, 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T07:26:20.018 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: "rules": [ 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_id": 0, 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_name": "replicated_rule", 2024-04-25T07:26:20.019 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: "steps": [ 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "take", 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: "item": -1, 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: "item_name": "default" 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "choose_firstn", 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 0, 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "osd" 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "emit" 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:20.020 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_id": 1, 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_name": "test", 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 3, 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: "steps": [ 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "set_chooseleaf_tries", 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 5 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "set_choose_tries", 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 100 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.021 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "take", 2024-04-25T07:26:20.022 INFO:tasks.workunit.client.0.smithi102.stdout: "item": -1, 2024-04-25T07:26:20.022 INFO:tasks.workunit.client.0.smithi102.stdout: "item_name": "default" 2024-04-25T07:26:20.022 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.022 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.022 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "choose_indep", 2024-04-25T07:26:20.022 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 0, 2024-04-25T07:26:20.022 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "osd" 2024-04-25T07:26:20.022 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.022 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:report 2272062396 2024-04-25T07:26:20.023 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "emit" 2024-04-25T07:26:20.023 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.023 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T07:26:20.023 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.023 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:20.023 INFO:tasks.workunit.client.0.smithi102.stdout: "tunables": { 2024-04-25T07:26:20.023 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_tries": 0, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_total_tries": 50, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_stable": 1, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_descents": 100, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_collision_tries": 100, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "straw_calc_version": 1, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "profile": "jewel", 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_tunables": 1, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "legacy_tunables": 0, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "minimum_required_version": "jewel", 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables": 1, 2024-04-25T07:26:20.024 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables2": 1, 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v2_rules": 1, 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables3": 1, 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v3_rules": 0, 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v4_buckets": 1, 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables5": 1, 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v5_rules": 0, 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "has_msr_rules": 0 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_args": {} 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "fsmap": { 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "default_fscid": -1, 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": { 2024-04-25T07:26:20.025 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": {}, 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "ro_compat": {}, 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "incompat": { 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_1": "base v0.20", 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_8": "no anchor table", 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_9": "file layout v2", 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_11": "minor log segments", 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.026 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_flags": { 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "enable_multiple": true, 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "ever_enabled_multiple": true 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "standbys": [], 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "filesystems": [] 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "auth": { 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 5, 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "num_secrets": 3 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.027 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 1, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_active": 1, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_unknown": 0, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osd": 3, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_sum": { 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 62667, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 101, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 1, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 303, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 101, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 101, 2024-04-25T07:26:20.028 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 101, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 101, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:26:20.029 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:26:20.030 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 0, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 0, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 18, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 18, 2024-04-25T07:26:20.031 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 3, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 3, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 0 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum": { 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 3, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 85104, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2300, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 15, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 82736, 2024-04-25T07:26:20.032 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314487696, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322035400704, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2355200, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 1611067, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 16263, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 84721785 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:26:20.033 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.034 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum_by_class": { 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "hdd": { 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 3, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 3, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 3, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 314572800, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 85104, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 2300, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 15, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 82736, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 314487696, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 322122547200, 2024-04-25T07:26:20.035 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 322035400704, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 2355200, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 1611067, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 16263, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 84721785 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 0, 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T07:26:20.036 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [], 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 1 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_stats": [ 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.037 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 1, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 62667, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 101, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 1, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 303, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 101, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 101, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T07:26:20.038 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 101, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 101, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T07:26:20.039 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T07:26:20.040 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 1241088, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 1241088, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 18, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 18, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 3, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 3, 2024-04-25T07:26:20.041 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 3 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_stats": [ 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 197568495626 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 42949672993 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 214748364810 2024-04-25T07:26:20.042 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_state": [ 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "active+undersized+degraded", 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 1 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_osd": [ 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2147483647, 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T07:26:20.043 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 1, 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 1, 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 0, 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 0, 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:20.044 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": [ 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": {} 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: "servicemap": { 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 5, 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T07:24:35.641582+0000", 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: "services": {} 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_last_committed": 128, 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_first_committed": 1, 2024-04-25T07:26:20.045 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_last_committed": 488, 2024-04-25T07:26:20.046 INFO:tasks.workunit.client.0.smithi102.stdout: "paxos": { 2024-04-25T07:26:20.046 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T07:26:20.046 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 674, 2024-04-25T07:26:20.046 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pn": 0, 2024-04-25T07:26:20.046 INFO:tasks.workunit.client.0.smithi102.stdout: "accepted_pn": 0 2024-04-25T07:26:20.046 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T07:26:20.046 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:26:20.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:26:20.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-04-25T07:26:20.047 INFO:tasks.workunit.client.0.smithi102.stdout:delay recovery 2024-04-25T07:26:20.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:26:20.375 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:26:20.375 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:25:44.069358+0000 43'102 52:141 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:26:20.375 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:26:20.375 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:26:20.376 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:26:20.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:26:20.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-04-25T07:26:20.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:26:20.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:26:20.392 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:26:20.392 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:20.392 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:26:20.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.0.asok 2024-04-25T07:26:20.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:26:20.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:26:20.497 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:26:20.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:26:20.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-04-25T07:26:20.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:26:20.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:26:20.510 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:26:20.510 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:20.510 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:26:20.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.2.asok 2024-04-25T07:26:20.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:26:20.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:26:20.617 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:26:20.629 INFO:tasks.workunit.client.0.smithi102.stdout:revive divergent 1 2024-04-25T07:26:20.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-04-25T07:26:20.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:26:21.036 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:26:21.036 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:25:44.069358+0000 43'102 52:141 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:26:21.036 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:26:21.036 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:26:21.037 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:26:21.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-04-25T07:26:21.744 INFO:tasks.workunit.client.0.smithi102.stderr:noup is set 2024-04-25T07:26:21.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-04-25T07:26:21.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:26:21.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:26:21.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:26:21.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:26:21.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:26:21.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:26:21.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:21.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:21.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:21.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:26:21.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:26:21.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:21.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:26:21.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:21.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:21.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:21.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:21.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:26:21.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:26:21.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:26:21.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:26:21.769 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:26:21.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:26:21.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:26:21.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:26:21.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:26:21.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:26:21.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:26:21.809 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:21.812+0000 7f58afa8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:21.836 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:21.836+0000 7f58afa8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:21.837 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:21.840+0000 7f58afa8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:22.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:26:22.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-04-25T07:26:22.371 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:22.372+0000 7f58afa8c7c0 -1 Falling back to public interface 2024-04-25T07:26:24.969 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:24.972+0000 7f58afa8c7c0 -1 osd.1 29 log_to_monitors true 2024-04-25T07:26:27.248 INFO:tasks.workunit.client.0.smithi102.stdout:delay recovery divergent 2024-04-25T07:26:27.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-04-25T07:26:27.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:26:27.650 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:26:27.650 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:25:44.069358+0000 43'102 54:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:33:43.247885+0000 0 0 2024-04-25T07:26:27.650 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:26:27.650 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:26:27.650 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:26:27.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-04-25T07:26:27.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:26:27.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:26:27.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:26:27.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:27.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:26:27.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26156/ceph-osd.1.asok 2024-04-25T07:26:27.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:26:27.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26156/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:26:27.781 INFO:tasks.workunit.client.0.smithi102.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:26:27.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-04-25T07:26:28.572 INFO:tasks.workunit.client.0.smithi102.stderr:noup is unset 2024-04-25T07:26:28.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-04-25T07:26:28.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:28.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:26:28.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:28.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:28.592 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:26:28.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:28.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:28.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:28.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:29.182 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 46 up_thru 51 down_at 45 last_clean_interval [36,43) [v2:127.0.0.1:6802/2563715480,v1:127.0.0.1:6803/2563715480] [v2:127.0.0.1:6804/2563715480,v1:127.0.0.1:6805/2563715480] exists,up 87e4e2ff-6f89-4a5e-9a0b-7ee1f6361e7e 2024-04-25T07:26:29.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:29.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:29.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:29.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-04-25T07:26:29.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:29.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:26:29.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:29.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:29.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:29.183 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:26:29.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:29.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:26:29.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:29.593 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/4126346999,v1:127.0.0.1:6819/4126346999] [v2:127.0.0.1:6820/4126346999,v1:127.0.0.1:6821/4126346999] exists,up 7e6be8e7-02ca-4c7c-a7ae-17fd198b434f 2024-04-25T07:26:29.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:29.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:29.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:29.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-04-25T07:26:29.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:29.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:26:29.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:29.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:29.595 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:26:29.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:29.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:29.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:29.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:26:30.056 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [38,43) [v2:127.0.0.1:6810/3198035605,v1:127.0.0.1:6811/3198035605] [v2:127.0.0.1:6812/3198035605,v1:127.0.0.1:6813/3198035605] exists,up 58d268a7-eec2-4e03-8fee-7633b0a83ab6 2024-04-25T07:26:30.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:30.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:30.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:30.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:26:30.450 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:26:30.450 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:25:44.069358+0000 43'102 54:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:33:43.247885+0000 0 0 2024-04-25T07:26:30.450 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:26:30.450 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:26:30.451 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:26:30.465 INFO:tasks.workunit.client.0.smithi102.stdout:wait for peering 2024-04-25T07:26:30.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-04-25T07:26:30.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:26:30.867 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:26:30.868 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:25:44.069358+0000 43'102 54:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:33:43.247885+0000 0 0 2024-04-25T07:26:30.868 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:26:30.868 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:26:30.868 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:26:30.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-04-25T07:26:31.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-04-25T07:26:31.042 INFO:tasks.workunit.client.0.smithi102.stdout:killing divergent 1 2024-04-25T07:26:31.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:26:31.449 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:26:31.449 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:25:44.069358+0000 43'102 54:145 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:33:43.247885+0000 0 0 2024-04-25T07:26:31.449 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:26:31.449 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:26:31.449 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:26:31.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:26:31.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:31.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:31.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:31.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:31.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:31.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:31.574 INFO:tasks.workunit.client.0.smithi102.stdout:reviving divergent 1 2024-04-25T07:26:31.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-04-25T07:26:31.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:26:31.987 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:26:31.987 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-04-25T07:26:29.731338+0000 43'102 58:135 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:26:31.988 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:26:31.988 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:26:31.988 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:26:32.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-04-25T07:26:32.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:26:32.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:26:32.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:26:32.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:26:32.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:26:32.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:26:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' 2024-04-25T07:26:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:26:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:26:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:26:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:26:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:26:32.010 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:26:32.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:26:32.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=5aac55d2-2692-43c2-a04d-c7f10efd07d7 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26156/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:26:32.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:26:32.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:26:32.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:26:32.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:26:32.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:26:32.044 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:32.047+0000 7f596d61c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:32.046 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:32.051+0000 7f596d61c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:32.047 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:32.051+0000 7f596d61c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:32.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:26:32.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:32.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:26:32.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:32.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:32.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:32.453 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:26:32.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:32.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:26:32.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:32.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:32.623+0000 7f596d61c7c0 -1 Falling back to public interface 2024-04-25T07:26:32.872 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/4126346999,v1:127.0.0.1:6819/4126346999] [v2:127.0.0.1:6820/4126346999,v1:127.0.0.1:6821/4126346999] exists,up 7e6be8e7-02ca-4c7c-a7ae-17fd198b434f 2024-04-25T07:26:32.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:32.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:32.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:32.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-04-25T07:26:35.893 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:35.899+0000 7f596d61c7c0 -1 osd.1 58 log_to_monitors true 2024-04-25T07:26:52.874 INFO:tasks.workunit.client.0.smithi102.stdout:allowing recovery 2024-04-25T07:26:52.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-04-25T07:26:52.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:26:53.290 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:26:53.290 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-04-25T07:26:38.125473+0000 58'103 63:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:06:53.302421+0000 0 0 2024-04-25T07:26:53.291 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:26:53.291 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:26:53.291 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:26:53.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:26:53.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:26:53.459 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:26:53.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:26:53.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:26:53.613 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:26:53.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:26:53.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:26:53.783 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:26:53.797 INFO:tasks.workunit.client.0.smithi102.stdout:reading divergent objects 2024-04-25T07:26:53.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-04-25T07:26:53.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-04-25T07:26:38.125473+0000 58'103 63:159 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:23:38.247919+0000 0'0 2024-04-25T07:23:38.247919+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:06:53.302421+0000 0 0 2024-04-25T07:26:54.188 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:26:54.189 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T07:26:54.205 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-04-25T07:26:54.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-04-25T07:26:54.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:26:54.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:26:54.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:26:54.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:26:54.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:26:54.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:26:54.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:26:54.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:26:54.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-04-25T07:26:54.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-04-25T07:26:54.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-04-25T07:24:20.121561+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 20 _merge_object_divergent_entries: keeping 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-04-25T07:24:20.121561+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 28'99 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4657.0:1 2024-04-25T07:24:26.817633+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 20 _merge_object_divergent_entries: keeping 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4657.0:1 2024-04-25T07:24:26.817633+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 28'99 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-04-25T07:26:54.383 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:26:54.384 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 29'104 2024-04-25T07:26:54.384 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4657.0:1 2024-04-25T07:24:26.817633+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:26:54.384 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-04-25T07:26:54.385 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4648.0:1 2024-04-25T07:24:25.586228+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false,28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4654.0:1 2024-04-25T07:24:26.590861+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:26:54.385 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4648.0:1 2024-04-25T07:24:25.586228+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:26:54.385 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 20 _merge_object_divergent_entries: keeping 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4654.0:1 2024-04-25T07:24:26.590861+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:26:54.385 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:26:54.385 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'102 2024-04-25T07:26:54.385 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-04-25T07:26:54.385 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:26:54.385 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'104 2024-04-25T07:26:54.385 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4654.0:1 2024-04-25T07:24:26.590861+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:26:54.385 INFO:tasks.workunit.client.0.smithi102.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:26:29.648+0000 7f5893a16640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-04-25T07:26:54.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-04-25T07:26:54.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:26:54.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-04-25T07:26:54.392 INFO:tasks.workunit.client.0.smithi102.stdout:success 2024-04-25T07:26:54.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-04-25T07:26:54.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:26:54.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:26:54.898 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T07:26:54.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-04-25T07:26:54.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:54.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:54.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:54.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:54.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:55.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:55.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:26:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:26:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:26:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:26:55.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:55.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:55.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:55.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:26:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:26:55.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:26:55.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:26:55.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:26:55.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:26:55.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:55.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:26:55.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:26:55.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:55.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:26:55.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:26:55.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:26:55.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:26:55.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:55.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:26:55.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26156 2024-04-25T07:26:55.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:26:55.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:26:55.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:26:55.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/divergent-priors 0 2024-04-25T07:26:55.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:26:55.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:26:55.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:26:55.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:55.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:55.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:55.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:55.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:55.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:55.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:26:55.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:26:55.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:26:55.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:26:55.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:26:55.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:26:55.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:55.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:26:55.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:26:55.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:55.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:26:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:26:55.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:26:55.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:26:55.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:55.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26156 2024-04-25T07:26:55.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26156 2024-04-25T07:26:55.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:26:55.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:26:55.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:26:55.070 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:26:55.071 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:26:55.080 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-04-25T07:26:55.081 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-04-25T07:26:55.132 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:26:55.136 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:26:55.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/ec-error-rollforward 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-04-25T07:26:55.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-04-25T07:26:55.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-04-25T07:26:55.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none ' 2024-04-25T07:26:55.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-04-25T07:26:55.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-04-25T07:26:55.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-04-25T07:26:55.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-04-25T07:26:55.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-04-25T07:26:55.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-04-25T07:26:55.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-04-25T07:26:55.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-04-25T07:26:55.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:26:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-04-25T07:26:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-04-25T07:26:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-04-25T07:26:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2024-04-25T07:26:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2024-04-25T07:26:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:26:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:26:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:26:55.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:55.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:55.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:55.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:55.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:55.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:26:55.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:26:55.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:26:55.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:26:55.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:26:55.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:26:55.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:55.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:26:55.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:26:55.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:26:55.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:26:55.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:26:55.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:26:55.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:26:55.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:55.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:26:55.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83856 2024-04-25T07:26:55.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:26:55.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:26:55.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2024-04-25T07:26:55.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:26:55.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:55.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:26:55.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83856 2024-04-25T07:26:55.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:26:55.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2024-04-25T07:26:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-04-25T07:26:55.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:26:55.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:55.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:55.646 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:55.646 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:55.646 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:26:55.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:26:55.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:26:55.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:26:55.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:26:55.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:26:55.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:26:55.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:26:55.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:26:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:26:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:26:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:26:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:26:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:26:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83856/ceph-mon.a.asok 2024-04-25T07:26:55.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:26:55.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83856/ceph-mon.a.asok config get fsid 2024-04-25T07:26:55.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:26:55.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:26:55.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:26:55.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:26:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:26:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:26:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:26:55.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:26:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:26:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:26:55.985 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83856/ceph-mon.a.asok 2024-04-25T07:26:55.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:26:55.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83856/ceph-mon.a.asok config get mon_host 2024-04-25T07:26:56.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-04-25T07:26:56.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2024-04-25T07:26:56.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:26:56.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:26:56.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:26:56.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2024-04-25T07:26:56.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:26:56.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:26:56.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:56.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:56.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:56.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:56.995 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:26:56.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:26:56.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:26:56.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:26:57.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-04-25T07:26:57.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:26:57.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:26:57.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:26:57.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:26:57.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2024-04-25T07:26:57.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:26:57.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:57.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:57.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:57.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:26:57.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2024-04-25T07:26:57.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:26:57.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=76d8968b-9b19-4980-af5b-d00d9a30d379 2024-04-25T07:26:57.034 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 76d8968b-9b19-4980-af5b-d00d9a30d379 2024-04-25T07:26:57.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 76d8968b-9b19-4980-af5b-d00d9a30d379' 2024-04-25T07:26:57.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:26:57.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBBSpmeoEwAxAAaTWogGrwR+kBLFyThGwXjQ== 2024-04-25T07:26:57.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBBSpmeoEwAxAAaTWogGrwR+kBLFyThGwXjQ=="}' 2024-04-25T07:26:57.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 76d8968b-9b19-4980-af5b-d00d9a30d379 -i td/ec-error-rollforward/0/new.json 2024-04-25T07:26:57.318 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:26:57.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2024-04-25T07:26:57.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBBSpmeoEwAxAAaTWogGrwR+kBLFyThGwXjQ== --osd-uuid 76d8968b-9b19-4980-af5b-d00d9a30d379 2024-04-25T07:26:57.358 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:57.363+0000 7fecf02137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:57.386 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:57.391+0000 7fecf02137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:57.388 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:57.391+0000 7fecf02137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:57.388 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:57.391+0000 7fecf02137c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:26:57.388 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:57.391+0000 7fecf02137c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:26:57.388 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:57.391+0000 7fecf02137c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:26:57.426 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:26:57.431+0000 7fecf02137c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-04-25T07:27:03.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-04-25T07:27:03.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:27:03.697 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:27:03.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:27:03.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:27:04.246 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:27:04.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:27:04.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:27:04.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:27:04.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:27:04.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:27:04.275 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:04.279+0000 7f5509a4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:04.277 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:04.283+0000 7f5509a4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:04.278 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:04.283+0000 7f5509a4c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:04.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:27:04.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:27:04.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:27:04.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:27:04.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:27:04.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:04.731 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:27:04.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:27:04.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:04.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:04.850 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:04.855+0000 7f5509a4c7c0 -1 Falling back to public interface 2024-04-25T07:27:05.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:06.195 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:27:06.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:06.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:06.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:27:06.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:06.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:06.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:07.382 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:07.387+0000 7f5509a4c7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:27:07.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:07.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:07.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:27:07.599 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:27:07.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:07.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:08.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:09.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:09.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:09.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:27:09.128 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:27:09.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:09.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:09.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:10.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:10.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:10.593 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:27:10.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:27:10.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:10.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:11.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:12.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:12.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:12.027 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:27:12.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:27:12.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:12.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:12.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:13.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:13.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:13.560 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:27:13.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:27:13.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:13.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:13.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:14.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:14.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:14.992 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:27:14.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:27:14.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:14.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:15.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:16.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:16.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:16.514 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:27:16.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:27:16.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:16.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:16.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:17.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:17.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:17.951 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:27:17.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:27:17.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:17.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:18.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:27:19.379 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:27:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:19.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:20.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:20.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:20.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:27:20.797 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:27:20.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:20.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:21.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:22.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:22.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:22.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:27:22.225 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:27:22.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:22.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:23.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:24.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:24.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:24.092 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:27:24.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:27:24.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:24.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:24.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:25.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:25.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:25.937 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:27:25.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:27:25.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:25.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:26.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:27.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:27.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:27.875 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:27:27.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:27:27.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:27.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:28.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:29.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:29.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:29.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:27:29.930 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:27:29.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:29.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:31.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:32.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:32.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:32.073 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:27:32.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:27:32.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:32.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:32.568 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/541369329,v1:127.0.0.1:6803/541369329] [v2:127.0.0.1:6804/541369329,v1:127.0.0.1:6805/541369329] exists,up 76d8968b-9b19-4980-af5b-d00d9a30d379 2024-04-25T07:27:32.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:27:32.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:27:32.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:27:32.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-04-25T07:27:32.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:27:32.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:27:32.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:27:32.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:32.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:32.572 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:32.572 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:32.572 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:27:32.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:27:32.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:27:32.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:27:32.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:27:32.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:27:32.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:27:32.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:27:32.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:27:32.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:27:32.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:27:32.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:27:32.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:27:32.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:27:32.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2024-04-25T07:27:32.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:27:32.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0833c4f9-2afe-4a09-9353-0829c0532fd1 2024-04-25T07:27:32.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0833c4f9-2afe-4a09-9353-0829c0532fd1' 2024-04-25T07:27:32.574 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 0833c4f9-2afe-4a09-9353-0829c0532fd1 2024-04-25T07:27:32.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:27:32.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkBSpmVAq5IxAA0ZrWoD/mTYg2LsRwViQdIQ== 2024-04-25T07:27:32.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkBSpmVAq5IxAA0ZrWoD/mTYg2LsRwViQdIQ=="}' 2024-04-25T07:27:32.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0833c4f9-2afe-4a09-9353-0829c0532fd1 -i td/ec-error-rollforward/1/new.json 2024-04-25T07:27:33.048 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:27:33.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2024-04-25T07:27:33.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkBSpmVAq5IxAA0ZrWoD/mTYg2LsRwViQdIQ== --osd-uuid 0833c4f9-2afe-4a09-9353-0829c0532fd1 2024-04-25T07:27:33.090 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:33.095+0000 7fd134c627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:33.102 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:33.107+0000 7fd134c627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:33.103 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:33.107+0000 7fd134c627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:33.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:33.107+0000 7fd134c627c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:27:33.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:33.107+0000 7fd134c627c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:27:33.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:33.107+0000 7fd134c627c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:27:33.133 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:33.139+0000 7fd134c627c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-04-25T07:27:39.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-04-25T07:27:39.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:27:39.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:27:39.764 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:27:39.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:27:40.320 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:27:40.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:27:40.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:27:40.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:27:40.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:27:40.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:27:40.350 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:40.355+0000 7f26cb8437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:40.368 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:40.375+0000 7f26cb8437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:40.369 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:40.375+0000 7f26cb8437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:40.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:27:40.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:27:40.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:27:40.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:27:40.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:27:40.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:40.745 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:27:40.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:27:40.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:40.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:40.925 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:40.931+0000 7f26cb8437c0 -1 Falling back to public interface 2024-04-25T07:27:41.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:42.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:42.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:42.168 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:27:42.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:27:42.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:42.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:42.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:43.559 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:27:43.562+0000 7f26cb8437c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:27:43.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:43.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:43.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:27:43.574 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:27:43.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:43.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:44.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:45.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:45.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:45.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:27:45.139 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:27:45.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:45.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:45.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:46.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:46.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:46.659 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:27:46.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:27:46.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:46.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:47.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:48.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:48.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:48.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:27:48.095 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:27:48.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:48.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:48.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:49.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:49.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:49.503 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:27:49.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:27:49.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:49.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:49.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:50.986 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:27:50.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:27:50.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:50.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:51.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:52.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:52.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:52.492 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:27:52.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:27:52.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:52.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:53.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:54.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:54.019 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:27:54.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:54.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:27:54.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:54.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:54.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:55.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:55.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:55.485 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:27:55.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:27:55.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:55.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:55.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:56.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:56.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:56.893 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:27:56.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:27:56.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:56.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:57.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:58.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:58.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:58.639 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:27:58.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:27:58.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:58.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:59.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:00.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:00.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:00.355 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:28:00.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:28:00.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:00.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:01.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:02.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:02.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:02.481 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:28:02.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:28:02.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:02.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:03.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:04.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:04.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:04.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:28:04.229 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:28:04.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:04.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:05.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:06.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:06.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:06.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:28:06.044 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:28:06.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:06.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:07.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:08.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:08.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:08.236 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:28:08.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:28:08.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:08.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:08.730 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3580448061,v1:127.0.0.1:6811/3580448061] [v2:127.0.0.1:6812/3580448061,v1:127.0.0.1:6813/3580448061] exists,up 0833c4f9-2afe-4a09-9353-0829c0532fd1 2024-04-25T07:28:08.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:28:08.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:28:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:28:08.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:28:08.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:28:08.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:28:08.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:28:08.734 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:28:08.734 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:08.734 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:28:08.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:28:08.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:28:08.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:28:08.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:28:08.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:28:08.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:28:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:28:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:28:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:28:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:28:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:28:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:28:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:28:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2024-04-25T07:28:08.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:28:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e616d4f6-bc28-4901-9b1f-dc5771eaeb6c 2024-04-25T07:28:08.737 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 e616d4f6-bc28-4901-9b1f-dc5771eaeb6c 2024-04-25T07:28:08.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e616d4f6-bc28-4901-9b1f-dc5771eaeb6c' 2024-04-25T07:28:08.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:28:08.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAIBipmtAI0LRAABNYUPQjOyam+ydKMMS+xrA== 2024-04-25T07:28:08.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAIBipmtAI0LRAABNYUPQjOyam+ydKMMS+xrA=="}' 2024-04-25T07:28:08.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e616d4f6-bc28-4901-9b1f-dc5771eaeb6c -i td/ec-error-rollforward/2/new.json 2024-04-25T07:28:09.295 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:28:09.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2024-04-25T07:28:09.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIBipmtAI0LRAABNYUPQjOyam+ydKMMS+xrA== --osd-uuid e616d4f6-bc28-4901-9b1f-dc5771eaeb6c 2024-04-25T07:28:09.338 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:09.342+0000 7fc9779a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:09.369 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:09.374+0000 7fc9779a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:09.370 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:09.374+0000 7fc9779a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:09.371 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:09.374+0000 7fc9779a37c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:28:09.371 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:09.374+0000 7fc9779a37c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:28:09.371 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:09.374+0000 7fc9779a37c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:28:09.417 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:09.422+0000 7fc9779a37c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-04-25T07:28:16.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-04-25T07:28:16.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:28:16.032 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:28:16.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:28:16.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:28:16.617 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:28:16.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:28:16.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:28:16.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:28:16.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:28:16.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:28:16.646 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:16.650+0000 7f5fe4c1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:16.649 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:16.654+0000 7f5fe4c1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:16.650 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:16.654+0000 7f5fe4c1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:17.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:28:17.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:28:17.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:28:17.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:28:17.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:28:17.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:17.079 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:28:17.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:28:17.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:17.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:17.226 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:17.230+0000 7f5fe4c1b7c0 -1 Falling back to public interface 2024-04-25T07:28:17.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:18.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:18.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:18.518 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:28:18.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:28:18.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:18.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:18.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:19.893 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:19.898+0000 7f5fe4c1b7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:28:19.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:19.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:19.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:28:19.934 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:28:19.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:19.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:20.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:21.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:21.697 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:28:21.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:21.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:28:21.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:21.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:22.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:23.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:23.388 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:28:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:28:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:24.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:25.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:25.008 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:28:25.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:25.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:28:25.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:25.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:25.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:26.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:26.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:26.465 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:28:26.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:28:26.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:26.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:27.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:28.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:28.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:28.013 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:28:28.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:28:28.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:28.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:28.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:29.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:29.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:29.625 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:28:29.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:28:29.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:29.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:30.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:31.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:31.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:31.142 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:28:31.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:28:31.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:31.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:31.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:32.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:32.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:32.579 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:28:32.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:28:32.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:32.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:33.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:34.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:34.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:34.048 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:28:34.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:28:34.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:34.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:35.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:36.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:36.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:36.128 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:28:36.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:28:36.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:36.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:36.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:37.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:37.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:37.810 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:28:37.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:28:37.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:37.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:39.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:40.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:40.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:40.161 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:28:40.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:28:40.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:40.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:40.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:41.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:41.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:41.898 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:28:41.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:28:41.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:41.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:43.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:44.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:44.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:44.171 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:28:44.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:28:44.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:44.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:45.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:46.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:46.062 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:28:46.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:46.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:28:46.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:46.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:46.469 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/432212043,v1:127.0.0.1:6819/432212043] [v2:127.0.0.1:6820/432212043,v1:127.0.0.1:6821/432212043] exists,up e616d4f6-bc28-4901-9b1f-dc5771eaeb6c 2024-04-25T07:28:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:28:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:28:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:28:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-04-25T07:28:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:28:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:28:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T07:28:46.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:28:46.473 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:28:46.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:28:46.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:28:46.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:28:46.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2024-04-25T07:28:46.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:28:46.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5f9a59c4-3bb1-4ca0-9683-c164e03823ae 2024-04-25T07:28:46.475 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 5f9a59c4-3bb1-4ca0-9683-c164e03823ae 2024-04-25T07:28:46.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5f9a59c4-3bb1-4ca0-9683-c164e03823ae' 2024-04-25T07:28:46.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:28:46.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuBipmbI7NHRAAW7wPbdnc2WNnKQUQNzgGbg== 2024-04-25T07:28:46.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuBipmbI7NHRAAW7wPbdnc2WNnKQUQNzgGbg=="}' 2024-04-25T07:28:46.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5f9a59c4-3bb1-4ca0-9683-c164e03823ae -i td/ec-error-rollforward/3/new.json 2024-04-25T07:28:46.987 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:28:47.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2024-04-25T07:28:47.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAuBipmbI7NHRAAW7wPbdnc2WNnKQUQNzgGbg== --osd-uuid 5f9a59c4-3bb1-4ca0-9683-c164e03823ae 2024-04-25T07:28:47.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:47.041+0000 7f275c5fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:47.077 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:47.081+0000 7f275c5fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:47.079 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:47.081+0000 7f275c5fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:47.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:47.081+0000 7f275c5fc7c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:28:47.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:47.081+0000 7f275c5fc7c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:28:47.080 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:47.081+0000 7f275c5fc7c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:28:47.123 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:47.125+0000 7f275c5fc7c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-04-25T07:28:53.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-04-25T07:28:53.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:28:53.671 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T07:28:53.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T07:28:53.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:28:54.226 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T07:28:54.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T07:28:54.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:28:54.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:28:54.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:28:54.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:28:54.256 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:54.257+0000 7f04695067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:54.299 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:54.301+0000 7f04695067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:54.301 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:54.305+0000 7f04695067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:54.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T07:28:54.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:28:54.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:28:54.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:28:54.691 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:28:54.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:28:54.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:54.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:28:54.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:54.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:28:54.879 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:54.881+0000 7f04695067c0 -1 Falling back to public interface 2024-04-25T07:28:55.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:56.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:56.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:56.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:28:56.092 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:28:56.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:56.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:28:56.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:57.392 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:28:57.393+0000 7f04695067c0 -1 osd.3 0 log_to_monitors true 2024-04-25T07:28:57.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:57.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:57.504 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:28:57.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:28:57.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:57.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:28:58.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:59.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:59.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:59.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:28:59.013 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:28:59.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:59.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:28:59.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:00.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:00.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:00.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:29:00.495 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:29:00.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:00.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:00.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:01.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:01.998 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:29:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:29:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:01.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:02.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:03.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:03.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:03.478 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:29:03.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:29:03.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:03.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:03.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:04.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:04.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:04.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:29:04.926 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:29:04.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:04.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:05.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:06.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:06.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:06.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:29:06.499 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:29:06.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:06.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:06.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:07.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:07.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:07.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:29:07.942 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:29:07.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:07.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:08.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:09.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:09.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:09.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:29:09.456 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:29:09.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:09.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:09.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:10.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:10.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:10.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:29:10.918 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:29:10.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:10.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:11.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:12.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:12.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:12.845 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:29:12.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:29:12.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:12.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:13.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:14.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:14.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:14.703 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:29:14.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:29:14.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:14.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:15.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:16.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:16.344 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:29:16.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:16.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:29:16.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:16.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:17.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:18.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:18.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:18.484 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:29:18.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:29:18.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:18.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:19.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:20.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:20.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:20.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:29:20.294 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:29:20.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:20.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:21.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:22.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:22.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:22.049 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:29:22.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:29:22.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:22.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:22.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:23.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:23.609 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:29:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:29:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:23.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:24.096 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2205588580,v1:127.0.0.1:6827/2205588580] [v2:127.0.0.1:6828/2205588580,v1:127.0.0.1:6829/2205588580] exists,up 5f9a59c4-3bb1-4ca0-9683-c164e03823ae 2024-04-25T07:29:24.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:29:24.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:29:24.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:29:24.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-04-25T07:29:24.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-04-25T07:29:25.469 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'ec' already exists 2024-04-25T07:29:25.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-04-25T07:29:27.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-04-25T07:29:27.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 88414 2024-04-25T07:29:27.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=91149 2024-04-25T07:29:27.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-04-25T07:29:27.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-04-25T07:29:28.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 91170' 2024-04-25T07:29:28.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-04-25T07:29:28.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 91171' 2024-04-25T07:29:28.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-04-25T07:29:28.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 91172' 2024-04-25T07:29:28.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-04-25T07:29:28.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-04-25T07:29:29.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-04-25T07:29:29.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:29:29.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:29:29.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:29:29.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:29:29.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:29:29.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:29:29.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 91149 91170 91171 91172 2024-04-25T07:29:29.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-04-25T07:29:29.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:29:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:29:29.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:29:29.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:29:29.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:29:29.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:29:29.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-04-25T07:29:29.728 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:29:29.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:29:29.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:29:29.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-04-25T07:29:29.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:29:29.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:29:29.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:29:29.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:29:29.753 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:29.756+0000 7efcb2be07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:29.755 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:29.756+0000 7efcb2be07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:29.757 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:29.760+0000 7efcb2be07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:30.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:29:30.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:29:30.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:29:30.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:29:30.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:29:30.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:30.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:29:30.198 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:29:30.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:30.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:30.328 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:30.328+0000 7efcb2be07c0 -1 Falling back to public interface 2024-04-25T07:29:30.642 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/541369329,v1:127.0.0.1:6803/541369329] [v2:127.0.0.1:6804/541369329,v1:127.0.0.1:6805/541369329] exists,up 76d8968b-9b19-4980-af5b-d00d9a30d379 2024-04-25T07:29:30.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:29:30.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:29:30.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:29:30.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-04-25T07:29:30.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:29:30.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:29:30.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:30.647 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:29:30.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:29:30.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-04-25T07:29:30.649 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:29:30.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:29:30.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:29:30.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-04-25T07:29:30.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:29:30.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:29:30.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:29:30.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:29:30.669 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:30.672+0000 7f8c0ebc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:30.694 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:30.696+0000 7f8c0ebc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:30.695 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:30.696+0000 7f8c0ebc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:31.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:29:31.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:29:31.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:29:31.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:29:31.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:29:31.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:31.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:29:31.042 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:29:31.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:31.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:29:31.250 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:31.252+0000 7f8c0ebc57c0 -1 Falling back to public interface 2024-04-25T07:29:31.465 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3580448061,v1:127.0.0.1:6811/3580448061] [v2:127.0.0.1:6812/3580448061,v1:127.0.0.1:6813/3580448061] exists,up 0833c4f9-2afe-4a09-9353-0829c0532fd1 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:29:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:29:31.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:29:31.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:29:31.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:29:31.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:29:31.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:29:31.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:29:31.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:29:31.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:29:31.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:29:31.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:29:31.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-04-25T07:29:31.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:29:31.472 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:29:31.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:29:31.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-04-25T07:29:31.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:29:31.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:29:31.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:29:31.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:29:31.498 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:31.500+0000 7f2aa4a107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:31.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:31.516+0000 7f2aa4a107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:31.515 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:31.516+0000 7f2aa4a107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:31.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:29:31.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:29:31.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:29:31.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:29:31.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:29:31.897 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:29:31.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:31.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:29:31.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:31.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:32.079 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:32.080+0000 7f2aa4a107c0 -1 Falling back to public interface 2024-04-25T07:29:32.299 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/432212043,v1:127.0.0.1:6819/432212043] [v2:127.0.0.1:6820/432212043,v1:127.0.0.1:6821/432212043] exists,up e616d4f6-bc28-4901-9b1f-dc5771eaeb6c 2024-04-25T07:29:32.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:29:32.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:29:32.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:29:32.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:29:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:29:32.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:29:32.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:29:32.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:29:32.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:29:32.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:29:32.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-04-25T07:29:32.304 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T07:29:32.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:29:32.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=3fbb5d57-9c37-4b96-8fed-e97805ee81cd --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83856/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:29:32.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-04-25T07:29:32.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:29:32.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:29:32.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:29:32.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:29:32.325 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:32.328+0000 7f00211e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:32.331 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:32.332+0000 7f00211e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:32.332 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:32.332+0000 7f00211e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:32.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:29:32.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:29:32.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:29:32.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:29:32.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:29:32.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:32.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:29:32.701 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:29:32.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:32.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:32.883 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:32.884+0000 7f00211e47c0 -1 Falling back to public interface 2024-04-25T07:29:33.271 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2205588580,v1:127.0.0.1:6827/2205588580] [v2:127.0.0.1:6828/2205588580,v1:127.0.0.1:6829/2205588580] exists,up 5f9a59c4-3bb1-4ca0-9683-c164e03823ae 2024-04-25T07:29:33.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:29:33.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:29:33.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:29:33.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-04-25T07:29:33.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:29:33.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:29:33.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:29:33.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:29:33.273 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:29:33.274 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:29:33.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:29:33.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:29:33.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:29:33.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:29:33.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:29:33.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:29:33.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:29:33.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:29:33.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:29:33.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:29:33.884 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:29:33.884 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T07:29:33.884 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T07:29:33.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:29:33.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:29:33.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:29:34.048 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:34.048+0000 7efcb2be07c0 -1 osd.0 26 log_to_monitors true 2024-04-25T07:29:35.590 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:35.596+0000 7f8c0ebc57c0 -1 osd.1 26 log_to_monitors true 2024-04-25T07:29:36.587 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:36.592+0000 7f2aa4a107c0 -1 osd.2 25 log_to_monitors true 2024-04-25T07:29:37.106 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:37.111+0000 7f00211e47c0 -1 osd.3 26 log_to_monitors true 2024-04-25T07:29:39.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051587 2024-04-25T07:29:39.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051587 2024-04-25T07:29:39.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587' 2024-04-25T07:29:39.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:29:39.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:29:39.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953475 2024-04-25T07:29:39.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953475 2024-04-25T07:29:39.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475' 2024-04-25T07:29:39.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:29:39.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:29:39.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855363 2024-04-25T07:29:39.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855363 2024-04-25T07:29:39.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-150323855363' 2024-04-25T07:29:39.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:29:39.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:29:39.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822659 2024-04-25T07:29:39.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822659 2024-04-25T07:29:39.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-150323855363 3-154618822659' 2024-04-25T07:29:39.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:29:39.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-124554051587 2024-04-25T07:29:39.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:29:39.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:29:39.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-124554051587 2024-04-25T07:29:39.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:29:39.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051587 2024-04-25T07:29:39.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 124554051587' 2024-04-25T07:29:39.587 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 124554051587 2024-04-25T07:29:39.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:29:40.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051587 -lt 124554051587 2024-04-25T07:29:40.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:29:40.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:29:40.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-137438953475 2024-04-25T07:29:40.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:29:40.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:29:40.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-137438953475 2024-04-25T07:29:40.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953475 2024-04-25T07:29:40.049 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 137438953475 2024-04-25T07:29:40.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 137438953475' 2024-04-25T07:29:40.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:29:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953475 -lt 137438953475 2024-04-25T07:29:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:29:40.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-150323855363 2024-04-25T07:29:40.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:29:40.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:29:40.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-150323855363 2024-04-25T07:29:40.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:29:40.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855363 2024-04-25T07:29:40.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2024-04-25T07:29:40.548 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 150323855363 2024-04-25T07:29:40.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:29:40.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855363 -lt 150323855363 2024-04-25T07:29:40.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:29:40.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-154618822659 2024-04-25T07:29:40.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:29:40.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:29:40.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-154618822659 2024-04-25T07:29:40.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:29:40.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822659 2024-04-25T07:29:40.964 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 154618822659 2024-04-25T07:29:40.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 154618822659' 2024-04-25T07:29:40.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:29:41.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 154618822659 2024-04-25T07:29:41.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:29:42.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:29:42.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:29:42.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822659 -lt 154618822659 2024-04-25T07:29:42.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:29:42.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:29:42.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:29:43.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:29:43.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:29:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:29:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:29:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:29:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:29:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:29:43.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:29:43.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:29:43.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:29:43.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:29:43.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:29:44.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:29:44.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:29:44.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:29:44.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:29:44.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:29:44.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:29:44.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:29:44.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:29:44.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:29:44.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:29:44.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:29:44.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:44.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:29:44.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:29:44.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:44.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:29:44.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:29:44.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:29:44.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:29:44.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:44.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:29:44.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83856 2024-04-25T07:29:44.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:29:44.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:29:44.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:29:44.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/ec-error-rollforward 0 2024-04-25T07:29:44.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:29:44.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:29:44.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:29:44.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:29:44.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:29:44.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:29:44.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:29:44.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:29:44.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:29:44.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:29:44.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:29:44.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:29:44.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:29:44.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:29:44.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:29:44.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:44.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:29:44.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:29:44.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:44.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:29:44.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:29:44.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:29:44.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:29:44.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:44.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83856 2024-04-25T07:29:44.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83856 2024-04-25T07:29:44.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:29:44.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:29:44.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:29:44.729 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:29:44.729 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:29:44.784 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-04-25T07:29:44.785 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-04-25T07:29:44.841 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:29:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:29:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:29:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:29:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:29:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:29:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:29:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:29:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:29:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:29:44.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-bench 2024-04-25T07:29:44.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-04-25T07:29:44.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-04-25T07:29:44.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-04-25T07:29:44.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-04-25T07:29:44.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-04-25T07:29:44.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-04-25T07:29:44.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=bda6186d-61f4-44d8-aa9a-7d2aaababfdd --auth-supported=none ' 2024-04-25T07:29:44.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-04-25T07:29:44.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-04-25T07:29:44.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-04-25T07:29:44.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:29:44.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:29:44.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:29:44.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:29:44.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:29:44.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:29:44.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:29:44.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:29:44.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:29:44.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:29:44.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:44.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:29:44.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:29:44.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:29:44.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:29:44.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:29:44.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:29:44.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:29:44.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:44.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93678 2024-04-25T07:29:44.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93678 2024-04-25T07:29:44.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:29:44.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:29:44.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2024-04-25T07:29:44.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:29:44.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:44.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93678 2024-04-25T07:29:44.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.93678 2024-04-25T07:29:44.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:29:44.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:29:44.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:29:44.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:29:44.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-04-25T07:29:44.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-04-25T07:29:44.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-04-25T07:29:44.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-04-25T07:29:44.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2024-04-25T07:29:44.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:29:44.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:29:44.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:29:44.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2024-04-25T07:29:44.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-04-25T07:29:45.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:29:45.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:45.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:45.350 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:45.350 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:45.350 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93678 2024-04-25T07:29:45.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93678/$cluster-$name.asok' 2024-04-25T07:29:45.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.93678/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:29:45.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:29:45.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:29:45.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:29:45.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:29:45.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:29:45.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:29:45.561 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:29:45.561 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:29:45.561 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:29:45.562 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:29:45.562 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:45.562 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93678 2024-04-25T07:29:45.562 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.93678/ceph-mon.a.asok 2024-04-25T07:29:45.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:29:45.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.93678/ceph-mon.a.asok config get fsid 2024-04-25T07:29:45.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:29:45.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:29:45.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:29:45.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:29:45.685 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:29:45.685 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:29:45.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:29:45.685 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:29:45.686 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:29:45.686 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:45.686 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93678 2024-04-25T07:29:45.686 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.93678/ceph-mon.a.asok 2024-04-25T07:29:45.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:29:45.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.93678/ceph-mon.a.asok config get mon_host 2024-04-25T07:29:45.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-04-25T07:29:45.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2024-04-25T07:29:45.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:29:45.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:29:45.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:29:45.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2024-04-25T07:29:45.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:29:46.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:29:46.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:46.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:46.700 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:46.700 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:46.700 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93678 2024-04-25T07:29:46.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93678/$cluster-$name.asok' 2024-04-25T07:29:46.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:29:46.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.93678/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:29:46.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-04-25T07:29:46.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2024-04-25T07:29:46.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:29:46.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:29:46.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:29:46.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2024-04-25T07:29:46.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=bda6186d-61f4-44d8-aa9a-7d2aaababfdd --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93678 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93678/$cluster-$name.asok' 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.93678/$cluster-$name.asok' 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:29:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:29:46.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:29:46.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2024-04-25T07:29:46.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:29:46.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=75542e71-80fa-4a39-a611-3cd91de44c32 2024-04-25T07:29:46.733 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 75542e71-80fa-4a39-a611-3cd91de44c32 2024-04-25T07:29:46.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 75542e71-80fa-4a39-a611-3cd91de44c32' 2024-04-25T07:29:46.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:29:46.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBqBipmF5rvLBAAOFK0UIjgHDPdhsE+GX4sxw== 2024-04-25T07:29:46.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBqBipmF5rvLBAAOFK0UIjgHDPdhsE+GX4sxw=="}' 2024-04-25T07:29:46.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 75542e71-80fa-4a39-a611-3cd91de44c32 -i td/osd-bench/0/new.json 2024-04-25T07:29:47.055 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:29:47.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2024-04-25T07:29:47.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=bda6186d-61f4-44d8-aa9a-7d2aaababfdd --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.93678/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqBipmF5rvLBAAOFK0UIjgHDPdhsE+GX4sxw== --osd-uuid 75542e71-80fa-4a39-a611-3cd91de44c32 2024-04-25T07:29:47.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:47.107+0000 7fa039df87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:47.124 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:47.127+0000 7fa039df87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:47.131+0000 7fa039df87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:47.131+0000 7fa039df87c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:29:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:47.131+0000 7fa039df87c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:29:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:47.131+0000 7fa039df87c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:29:47.163 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:47.167+0000 7fa039df87c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-04-25T07:29:53.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2024-04-25T07:29:53.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:29:53.494 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:29:53.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:29:53.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:29:54.041 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:29:54.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:29:54.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:29:54.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=bda6186d-61f4-44d8-aa9a-7d2aaababfdd --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.93678/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:29:54.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:29:54.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:29:54.071 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:54.075+0000 7f84ed2337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:54.073 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:54.079+0000 7f84ed2337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:54.075 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:54.079+0000 7f84ed2337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:54.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:29:54.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:29:54.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:29:54.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:29:54.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:29:54.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:54.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:29:54.449 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:29:54.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:54.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:54.615 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:54.619+0000 7f84ed2337c0 -1 Falling back to public interface 2024-04-25T07:29:54.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:55.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:55.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:55.845 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:29:55.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:29:55.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:55.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:56.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:57.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:57.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:57.243 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:29:57.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:29:57.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:57.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:57.249 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:29:57.255+0000 7f84ed2337c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:29:57.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:58.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:58.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:58.761 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:29:58.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:29:58.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:58.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:29:59.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:00.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:00.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:00.297 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:30:00.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:30:00.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:00.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:00.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:01.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:01.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:01.774 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:30:01.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:30:01.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:01.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:02.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:03.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:03.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:03.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:30:03.308 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:30:03.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:03.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:03.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:04.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:04.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:04.750 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:30:04.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:30:04.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:04.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:05.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:06.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:06.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:06.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:30:06.216 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:30:06.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:06.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:06.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:07.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:07.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:07.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:30:07.658 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:30:07.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:07.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:08.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:09.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:09.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:09.193 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:30:09.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:30:09.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:09.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:09.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:10.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:10.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:10.607 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:30:10.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:30:10.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:10.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:11.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:12.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:12.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:12.026 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:30:12.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:30:12.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:12.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:12.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:13.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:30:13.898 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:30:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:13.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:14.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:15.709 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:30:15.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:30:15.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:15.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:16.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:17.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:17.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:17.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:30:17.911 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:30:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:17.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:18.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:19.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:19.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:19.925 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:30:19.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:30:19.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:19.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:21.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:22.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:22.082 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:30:22.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:22.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:30:22.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:22.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:22.565 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/253973614,v1:127.0.0.1:6803/253973614] [v2:127.0.0.1:6804/253973614,v1:127.0.0.1:6805/253973614] exists,up 75542e71-80fa-4a39-a611-3cd91de44c32 2024-04-25T07:30:22.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:30:22.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:30:22.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:30:22.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-04-25T07:30:22.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-04-25T07:30:22.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-04-25T07:30:22.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-04-25T07:30:22.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-04-25T07:30:22.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-04-25T07:30:22.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-04-25T07:30:22.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-04-25T07:30:22.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-04-25T07:30:22.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-04-25T07:30:22.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-04-25T07:30:22.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-04-25T07:30:22.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-04-25T07:30:22.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:30:22.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:30:22.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=osd_bench_max_block_size 2024-04-25T07:30:22.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:30:22.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:30:22.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-04-25T07:30:22.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:30:22.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:30:22.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-04-25T07:30:22.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:30:22.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-04-25T07:30:22.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-04-25T07:30:22.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-04-25T07:30:22.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:30:22.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:30:22.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=bench_small_size_max_iops 2024-04-25T07:30:22.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:30:22.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:30:22.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-04-25T07:30:22.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:30:22.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:30:22.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-04-25T07:30:22.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:30:22.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-04-25T07:30:22.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-04-25T07:30:22.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-04-25T07:30:22.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:30:22.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:30:22.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-04-25T07:30:22.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:30:22.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:30:22.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-04-25T07:30:23.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:30:23.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:30:23.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-04-25T07:30:23.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:30:23.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-04-25T07:31:18.241 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:31:18.241 INFO:tasks.workunit.client.0.smithi102.stdout: "bytes_written": 1073741824, 2024-04-25T07:31:18.242 INFO:tasks.workunit.client.0.smithi102.stdout: "blocksize": 4194304, 2024-04-25T07:31:18.242 INFO:tasks.workunit.client.0.smithi102.stdout: "elapsed_sec": 26.654792005000001, 2024-04-25T07:31:18.242 INFO:tasks.workunit.client.0.smithi102.stdout: "bytes_per_sec": 40283256.526578173, 2024-04-25T07:31:18.242 INFO:tasks.workunit.client.0.smithi102.stdout: "iops": 9.604276782650512 2024-04-25T07:31:18.242 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:31:18.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-04-25T07:31:18.758 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:31:18.759 INFO:tasks.workunit.client.0.smithi102.stdout: "bytes_written": 10, 2024-04-25T07:31:18.759 INFO:tasks.workunit.client.0.smithi102.stdout: "blocksize": 14456, 2024-04-25T07:31:18.759 INFO:tasks.workunit.client.0.smithi102.stdout: "elapsed_sec": 0.033456284000000003, 2024-04-25T07:31:18.759 INFO:tasks.workunit.client.0.smithi102.stdout: "bytes_per_sec": 298.89751055436994, 2024-04-25T07:31:18.759 INFO:tasks.workunit.client.0.smithi102.stdout: "iops": 0.020676363486052153 2024-04-25T07:31:18.759 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:31:18.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-04-25T07:31:19.059 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T07:31:19.059 INFO:tasks.workunit.client.0.smithi102.stdout: "bytes_written": 1, 2024-04-25T07:31:19.059 INFO:tasks.workunit.client.0.smithi102.stdout: "blocksize": 14456, 2024-04-25T07:31:19.059 INFO:tasks.workunit.client.0.smithi102.stdout: "elapsed_sec": 0.032632478999999999, 2024-04-25T07:31:19.059 INFO:tasks.workunit.client.0.smithi102.stdout: "bytes_per_sec": 30.644316050889056, 2024-04-25T07:31:19.059 INFO:tasks.workunit.client.0.smithi102.stdout: "iops": 0.0021198337057892263 2024-04-25T07:31:19.060 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T07:31:19.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-04-25T07:31:19.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:31:19.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:31:19.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:31:19.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:31:19.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:31:19.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:31:19.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:31:19.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:31:19.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:31:19.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:31:19.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:31:19.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:31:19.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:31:19.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:31:19.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:31:19.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:19.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:31:19.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:31:19.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:19.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:31:19.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:31:19.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:31:19.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:31:19.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:19.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93678 2024-04-25T07:31:19.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93678 2024-04-25T07:31:19.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:31:19.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:31:19.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:31:19.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-bench 0 2024-04-25T07:31:19.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:31:19.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:31:19.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:31:19.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:31:19.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:31:19.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:31:19.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:31:19.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:31:19.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:31:19.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:31:19.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:31:19.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:31:19.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:31:19.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:31:19.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:31:19.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:19.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:31:19.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:31:19.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:19.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:31:19.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:31:19.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:31:19.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:31:19.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:19.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93678 2024-04-25T07:31:19.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93678 2024-04-25T07:31:19.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:31:19.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:31:19.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:31:19.240 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:31:19.241 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:31:19.293 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-04-25T07:31:19.293 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-04-25T07:31:19.349 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:31:19.355 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:31:19.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:31:19.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:31:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:31:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:31:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:31:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:31:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:31:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:31:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-bluefs-volume-ops 2024-04-25T07:31:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:31:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-04-25T07:31:19.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-04-25T07:31:19.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:31:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-04-25T07:31:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_bluestore2 2024-04-25T07:31:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_bluestore_expand' 2024-04-25T07:31:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:31:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:31:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:31:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:31:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:31:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:31:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:31:19.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:31:19.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:31:19.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:31:19.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:31:19.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:31:19.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:31:19.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:31:19.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:31:19.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:31:19.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:31:19.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:31:19.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:31:19.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:19.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:31:19.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:31:19.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:31:19.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:31:19.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:31:19.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:31:19.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:31:19.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:19.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:31:19.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95853 2024-04-25T07:31:19.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:31:19.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:31:19.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:31:19.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:31:19.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:19.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:31:19.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.95853 2024-04-25T07:31:19.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:31:19.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:31:19.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:31:19.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:31:19.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:31:19.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-04-25T07:31:19.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:31:19.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-04-25T07:31:19.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-04-25T07:31:19.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-04-25T07:31:19.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:31:19.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-04-25T07:31:19.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-04-25T07:31:19.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-04-25T07:31:19.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none ' 2024-04-25T07:31:19.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:31:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:31:19.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:31:19.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:31:19.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:31:19.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:31:19.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:19.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:31:19.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:31:19.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:31:20.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:31:20.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:31:20.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:31:20.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:31:20.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:31:20.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:31:20.085 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:31:20.085 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:31:20.085 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:31:20.085 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:31:20.086 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:20.086 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:31:20.086 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95853/ceph-mon.a.asok 2024-04-25T07:31:20.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:31:20.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95853/ceph-mon.a.asok config get fsid 2024-04-25T07:31:20.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:31:20.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:31:20.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:31:20.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:31:20.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:31:20.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:31:20.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:31:20.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:31:20.205 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:31:20.205 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:20.205 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:31:20.206 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95853/ceph-mon.a.asok 2024-04-25T07:31:20.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:31:20.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95853/ceph-mon.a.asok config get mon_host 2024-04-25T07:31:20.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:31:20.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:31:20.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:31:20.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:31:20.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:31:20.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:31:20.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:31:21.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:31:21.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:31:21.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:31:21.219 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:31:21.219 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:21.219 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:31:21.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:31:21.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:31:21.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:31:21.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:31:21.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:31:21.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:31:21.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:31:21.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:31:21.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:31:21.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:31:21.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:31:21.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:31:21.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:31:21.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:31:21.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62b1a515-f1de-4a1b-865e-deaaf5cd7f26 2024-04-25T07:31:21.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 62b1a515-f1de-4a1b-865e-deaaf5cd7f26' 2024-04-25T07:31:21.261 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 62b1a515-f1de-4a1b-865e-deaaf5cd7f26 2024-04-25T07:31:21.261 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:31:21.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJBipm9OHrEBAAhDa0w1k6tPrsF2/Xp2GiHA== 2024-04-25T07:31:21.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJBipm9OHrEBAAhDa0w1k6tPrsF2/Xp2GiHA=="}' 2024-04-25T07:31:21.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62b1a515-f1de-4a1b-865e-deaaf5cd7f26 -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:31:21.584 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:31:21.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:31:21.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJBipm9OHrEBAAhDa0w1k6tPrsF2/Xp2GiHA== --osd-uuid 62b1a515-f1de-4a1b-865e-deaaf5cd7f26 2024-04-25T07:31:21.633 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:21.637+0000 7f24ffe507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:21.653 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:21.653+0000 7f24ffe507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:21.655 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:21.657+0000 7f24ffe507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:21.655 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:21.657+0000 7f24ffe507c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:31:21.655 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:21.657+0000 7f24ffe507c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:31:21.655 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:21.657+0000 7f24ffe507c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:31:21.693 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:21.693+0000 7f24ffe507c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:31:31.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:31:31.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:31:31.596 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:31:31.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:31:31.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:31:32.156 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:31:32.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:31:32.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:31:32.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:31:32.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:31:32.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:31:32.186 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:32.189+0000 7fdde585d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:32.189 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:32.189+0000 7fdde585d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:32.190 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:32.193+0000 7fdde585d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:31:32.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:31:32.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:31:32.544 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:31:32.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:31:32.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:31:32.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:31:32.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:32.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:31:32.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:32.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:33.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:33.260 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:33.261+0000 7fdde585d7c0 -1 Falling back to public interface 2024-04-25T07:31:34.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:34.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:34.005 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:31:34.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:31:34.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:34.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:34.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:35.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:35.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:35.409 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:31:35.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:31:35.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:35.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:35.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:36.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:36.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:36.807 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:31:36.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:31:36.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:36.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:37.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:38.112 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:31:38.113+0000 7fdde585d7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:31:38.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:38.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:38.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:31:38.215 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:31:38.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:38.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:38.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:39.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:39.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:39.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:31:39.846 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:31:39.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:39.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:40.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:41.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:41.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:41.328 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:31:41.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:31:41.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:41.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:41.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:42.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:42.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:42.749 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:31:42.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:31:42.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:42.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:43.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:44.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:44.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:44.256 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:31:44.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:31:44.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:44.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:44.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:45.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:45.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:45.659 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:31:45.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:31:45.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:45.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:46.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:47.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:47.218 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:31:47.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:47.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:31:47.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:47.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:47.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:48.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:48.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:48.668 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:31:48.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:31:48.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:48.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:49.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:50.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:50.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:50.126 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:31:50.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:31:50.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:50.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:50.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:51.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:51.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:51.562 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:31:51.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:31:51.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:51.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:52.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:31:53.928 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:31:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:54.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:55.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:55.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:55.968 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:31:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:31:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:55.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:57.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:58.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:58.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:58.198 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:31:58.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:31:58.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:58.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:59.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:00.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:00.129 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:32:00.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:32:00.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:00.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:01.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:02.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:02.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:02.553 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:32:02.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:32:02.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:02.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:03.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:04.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:04.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:04.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:32:04.129 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T07:32:04.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:04.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:04.581 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/665922015,v1:127.0.0.1:6803/665922015] [v2:127.0.0.1:6804/665922015,v1:127.0.0.1:6805/665922015] exists,up 62b1a515-f1de-4a1b-865e-deaaf5cd7f26 2024-04-25T07:32:04.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:32:04.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:32:04.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:32:04.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:32:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=97314 2024-04-25T07:32:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:32:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:32:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:32:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:32:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:32:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:32:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:32:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:32:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:32:04.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:32:04.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:32:04.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:32:04.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:32:04.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:32:04.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:32:04.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:32:04.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:32:04.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:32:04.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:32:04.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:32:04.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:32:04.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:32:04.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:32:04.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:32:04.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:32:04.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=51efadca-5b36-49e3-b603-35075dc6e371 2024-04-25T07:32:04.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 51efadca-5b36-49e3-b603-35075dc6e371' 2024-04-25T07:32:04.589 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 51efadca-5b36-49e3-b603-35075dc6e371 2024-04-25T07:32:04.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:32:04.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0Bipm/DxSJBAAq1Rh2NqKdf1z+hX+nY/bIA== 2024-04-25T07:32:04.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0Bipm/DxSJBAAq1Rh2NqKdf1z+hX+nY/bIA=="}' 2024-04-25T07:32:04.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 51efadca-5b36-49e3-b603-35075dc6e371 -i td/osd-bluefs-volume-ops/1/new.json 2024-04-25T07:32:05.136 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:32:05.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-04-25T07:32:05.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0Bipm/DxSJBAAq1Rh2NqKdf1z+hX+nY/bIA== --osd-uuid 51efadca-5b36-49e3-b603-35075dc6e371 2024-04-25T07:32:05.185 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:05.184+0000 7f46edfdf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:05.226 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:05.228+0000 7f46edfdf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:05.228 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:05.228+0000 7f46edfdf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:05.229 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:05.228+0000 7f46edfdf7c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:32:05.229 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:05.228+0000 7f46edfdf7c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:32:05.229 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:05.228+0000 7f46edfdf7c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:32:05.273 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:05.272+0000 7f46edfdf7c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-04-25T07:32:15.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-04-25T07:32:15.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:32:15.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:32:15.969 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:32:15.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:32:16.529 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:32:16.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:32:16.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:32:16.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:32:16.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:32:16.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:32:16.559 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:16.560+0000 7f3a6fa0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:16.590 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:16.592+0000 7f3a6fa0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:16.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:16.592+0000 7f3a6fa0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:16.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:32:16.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:32:16.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:32:16.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:32:16.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:32:16.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:16.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:32:16.995 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:32:16.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:16.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:17.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:17.632 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:17.632+0000 7f3a6fa0e7c0 -1 Falling back to public interface 2024-04-25T07:32:18.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:18.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:18.414 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:32:18.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:32:18.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:18.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:18.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:19.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:19.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:19.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:32:19.820 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:32:19.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:19.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:20.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:21.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:21.221 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:32:21.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:21.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:32:21.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:21.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:21.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:22.548 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:22.548+0000 7f3a6fa0e7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:32:22.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:22.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:22.616 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:32:22.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:32:22.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:22.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:23.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:24.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:24.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:24.319 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:32:24.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:32:24.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:24.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:24.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:25.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:25.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:25.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:32:25.931 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:32:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:25.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:26.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:27.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:27.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:27.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:32:27.581 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:32:27.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:27.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:28.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:29.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:29.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:29.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:32:29.032 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:32:29.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:29.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:29.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:30.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:30.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:30.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:32:30.608 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:32:30.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:30.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:31.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:32.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:32.074 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:32:32.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:32.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:32:32.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:32.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:32.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:33.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:33.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:33.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:32:33.536 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:32:33.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:33.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:34.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:35.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:35.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:35.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:32:35.065 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:32:35.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:35.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:35.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:36.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:36.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:36.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:32:36.595 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:32:36.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:36.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:37.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:38.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:38.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:38.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:32:38.163 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:32:38.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:38.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:38.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:39.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:39.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:39.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:32:39.785 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:32:39.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:39.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:40.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:41.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:41.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:41.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:32:41.411 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:32:41.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:41.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:42.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:43.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:43.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:43.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:32:43.176 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:32:43.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:43.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:44.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:45.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:45.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:45.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:32:45.107 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:32:45.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:45.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:45.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:46.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:46.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:46.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:32:46.911 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T07:32:46.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:46.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:47.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:48.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:48.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:48.537 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T07:32:48.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:32:48.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:48.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:48.945 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1652055127,v1:127.0.0.1:6811/1652055127] [v2:127.0.0.1:6812/1652055127,v1:127.0.0.1:6813/1652055127] exists,up 51efadca-5b36-49e3-b603-35075dc6e371 2024-04-25T07:32:48.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:32:48.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:32:48.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:32:48.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:32:48.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=99855 2024-04-25T07:32:48.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:32:48.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:32:48.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:32:48.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:32:48.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:32:48.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:32:48.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:32:48.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:32:48.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:32:48.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:32:48.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:32:48.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=86c0edaf-9fed-4a03-a25d-5450ee8c066f 2024-04-25T07:32:48.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 86c0edaf-9fed-4a03-a25d-5450ee8c066f' 2024-04-25T07:32:48.955 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 86c0edaf-9fed-4a03-a25d-5450ee8c066f 2024-04-25T07:32:48.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:32:48.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAgBypmkiEfOhAA1kfMirpRW9TjpL1aQDFtjA== 2024-04-25T07:32:48.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAgBypmkiEfOhAA1kfMirpRW9TjpL1aQDFtjA=="}' 2024-04-25T07:32:48.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 86c0edaf-9fed-4a03-a25d-5450ee8c066f -i td/osd-bluefs-volume-ops/2/new.json 2024-04-25T07:32:49.454 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:32:49.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-04-25T07:32:49.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgBypmkiEfOhAA1kfMirpRW9TjpL1aQDFtjA== --osd-uuid 86c0edaf-9fed-4a03-a25d-5450ee8c066f 2024-04-25T07:32:49.503 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:49.507+0000 7f87688027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:49.521 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:49.527+0000 7f87688027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:49.523 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:49.527+0000 7f87688027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:49.523 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:49.527+0000 7f87688027c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:32:49.523 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:49.527+0000 7f87688027c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:32:49.523 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:49.527+0000 7f87688027c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:32:49.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:32:49.563+0000 7f87688027c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-04-25T07:33:00.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-04-25T07:33:00.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:33:00.355 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:33:00.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:33:00.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:33:00.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:33:00.912 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:33:00.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:33:00.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:33:00.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:33:00.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:33:00.942 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:00.947+0000 7f542d84c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:00.945 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:00.947+0000 7f542d84c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:00.946 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:00.951+0000 7f542d84c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:01.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:33:01.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:33:01.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:33:01.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:33:01.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:33:01.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:01.342 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:33:01.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:33:01.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:01.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:01.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:01.996 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:01.999+0000 7f542d84c7c0 -1 Falling back to public interface 2024-04-25T07:33:02.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:02.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:02.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:33:02.763 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:33:02.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:02.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:03.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:04.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:04.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:04.163 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:33:04.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:33:04.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:04.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:04.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:05.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:05.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:05.590 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:33:05.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:33:05.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:05.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:06.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:06.917 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:06.923+0000 7f542d84c7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:33:07.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:07.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:07.029 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:33:07.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:33:07.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:07.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:07.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:08.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:08.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:08.650 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:33:08.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:33:08.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:08.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:09.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:10.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:33:10.083 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:33:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:10.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:10.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:11.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:11.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:33:11.632 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:33:11.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:11.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:12.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:13.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:13.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:13.151 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:33:13.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:33:13.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:13.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:13.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:14.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:14.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:14.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:33:14.572 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:33:14.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:14.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:15.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:16.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:16.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:16.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:33:16.114 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:33:16.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:16.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:16.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:17.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:17.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:17.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:33:17.597 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:33:17.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:17.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:18.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:19.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:19.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:19.159 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:33:19.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:33:19.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:19.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:19.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:20.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:20.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:20.577 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:33:20.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:33:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:21.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:22.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:22.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:22.302 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:33:22.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:33:22.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:22.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:22.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:23.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:23.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:23.989 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:33:23.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:33:23.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:23.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:25.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:26.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:26.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:26.293 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:33:26.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:33:26.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:26.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:27.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:28.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:28.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:28.557 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:33:28.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:33:28.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:28.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:29.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:30.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:30.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:30.729 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:33:30.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:33:30.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:30.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:31.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:32.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:32.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:32.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:33:32.533 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T07:33:32.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:32.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:33.063 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/215512675,v1:127.0.0.1:6819/215512675] [v2:127.0.0.1:6820/215512675,v1:127.0.0.1:6821/215512675] exists,up 86c0edaf-9fed-4a03-a25d-5450ee8c066f 2024-04-25T07:33:33.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:33:33.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:33:33.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:33:33.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:33:33.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=102421 2024-04-25T07:33:33.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:33:33.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:33:33.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:33:33.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:33:33.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:33:33.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:33:33.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:33:33.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:33:33.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:33:33.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:33:33.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:33:33.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8271f3ee-32c9-4e2f-a4cb-99671ce33ed0 2024-04-25T07:33:33.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 8271f3ee-32c9-4e2f-a4cb-99671ce33ed0' 2024-04-25T07:33:33.072 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 8271f3ee-32c9-4e2f-a4cb-99671ce33ed0 2024-04-25T07:33:33.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:33:33.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNBypmlBh6BRAAby3znQ8NKJmCjDa0Pbxnvg== 2024-04-25T07:33:33.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNBypmlBh6BRAAby3znQ8NKJmCjDa0Pbxnvg=="}' 2024-04-25T07:33:33.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8271f3ee-32c9-4e2f-a4cb-99671ce33ed0 -i td/osd-bluefs-volume-ops/3/new.json 2024-04-25T07:33:33.559 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:33:33.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-04-25T07:33:33.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNBypmlBh6BRAAby3znQ8NKJmCjDa0Pbxnvg== --osd-uuid 8271f3ee-32c9-4e2f-a4cb-99671ce33ed0 2024-04-25T07:33:33.610 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:33.614+0000 7f8c51da77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:33.613 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:33.614+0000 7f8c51da77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:33.614 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:33.618+0000 7f8c51da77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:33.614 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:33.618+0000 7f8c51da77c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:33:33.614 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:33.618+0000 7f8c51da77c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:33:33.615 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:33.618+0000 7f8c51da77c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:33:33.676 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:33.678+0000 7f8c51da77c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-04-25T07:33:44.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-04-25T07:33:44.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:33:44.415 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T07:33:44.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T07:33:44.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:33:44.959 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T07:33:44.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T07:33:44.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:33:44.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:33:44.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:33:44.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:33:44.991 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:44.994+0000 7f8f42b787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:44.994 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:44.998+0000 7f8f42b787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:44.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:44.998+0000 7f8f42b787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:45.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T07:33:45.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:33:45.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:33:45.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:33:45.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:33:45.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:45.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:33:45.377 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:33:45.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:45.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:45.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:46.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:46.038+0000 7f8f42b787c0 -1 Falling back to public interface 2024-04-25T07:33:46.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:46.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:46.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:33:46.800 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:33:46.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:46.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:47.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:48.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:48.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:48.219 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:33:48.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:33:48.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:48.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:48.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:49.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:49.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:49.642 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:33:49.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:33:49.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:49.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:50.987 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:33:50.990+0000 7f8f42b787c0 -1 osd.3 0 log_to_monitors true 2024-04-25T07:33:51.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:51.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:51.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:33:51.064 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:33:51.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:51.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:51.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:52.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:52.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:52.684 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:33:52.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:33:52.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:52.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:53.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:54.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:54.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:54.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:33:54.285 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:33:54.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:54.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:54.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:55.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:55.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:55.765 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:33:55.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:33:55.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:55.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:57.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:57.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:57.209 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:33:57.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:33:57.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:57.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:57.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:58.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:58.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:58.621 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:33:58.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:33:58.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:58.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:33:59.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:00.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:00.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:00.225 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:34:00.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:34:00.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:00.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:00.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:01.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:01.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:01.764 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:34:01.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:34:01.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:01.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:02.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:03.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:03.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:03.264 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:34:03.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:34:03.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:03.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:03.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:04.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:04.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:04.850 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:34:04.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:34:04.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:04.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:05.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:06.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:06.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:06.277 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:34:06.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:34:06.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:06.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:06.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:07.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:07.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:07.911 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:34:07.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:34:07.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:07.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:08.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:09.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:09.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:09.413 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:34:09.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:34:09.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:09.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:10.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:11.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:11.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:11.224 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:34:11.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:34:11.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:11.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:11.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:12.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:12.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:12.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:34:12.931 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:34:12.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:12.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:13.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:14.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:14.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:14.725 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T07:34:14.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:34:14.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:14.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:15.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:16.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:16.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:16.607 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T07:34:16.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:34:16.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:16.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:17.248 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/940406781,v1:127.0.0.1:6827/940406781] [v2:127.0.0.1:6828/940406781,v1:127.0.0.1:6829/940406781] exists,up 8271f3ee-32c9-4e2f-a4cb-99671ce33ed0 2024-04-25T07:34:17.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:34:17.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:34:17.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:34:17.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:34:17.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=104963 2024-04-25T07:34:17.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-04-25T07:34:22.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-04-25T07:34:22.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:34:22.774 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T07:34:22.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:34:23.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:34:23.917 INFO:tasks.workunit.client.0.smithi102.stdout:hints = 1 2024-04-25T07:34:23.918 INFO:tasks.workunit.client.0.smithi102.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:34:23.918 INFO:tasks.workunit.client.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_106344 2024-04-25T07:34:23.918 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:34:23.918 INFO:tasks.workunit.client.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:34:24.918 INFO:tasks.workunit.client.0.smithi102.stdout: 1 16 45 29 0.113274 0.113281 0.336158 0.462134 2024-04-25T07:34:25.918 INFO:tasks.workunit.client.0.smithi102.stdout: 2 16 85 69 0.134749 0.15625 0.430898 0.403016 2024-04-25T07:34:26.918 INFO:tasks.workunit.client.0.smithi102.stdout: 3 16 131 115 0.14972 0.179688 0.316707 0.390789 2024-04-25T07:34:27.918 INFO:tasks.workunit.client.0.smithi102.stdout: 4 16 190 174 0.169901 0.230469 0.292149 0.355874 2024-04-25T07:34:28.918 INFO:tasks.workunit.client.0.smithi102.stdout: 5 16 238 222 0.173417 0.1875 0.291418 0.352995 2024-04-25T07:34:29.918 INFO:tasks.workunit.client.0.smithi102.stdout: 6 16 291 275 0.179015 0.207031 0.289279 0.341526 2024-04-25T07:34:30.918 INFO:tasks.workunit.client.0.smithi102.stdout: 7 16 340 324 0.180783 0.191406 0.305733 0.335758 2024-04-25T07:34:31.919 INFO:tasks.workunit.client.0.smithi102.stdout: 8 16 399 383 0.18699 0.230469 0.309235 0.32913 2024-04-25T07:34:32.919 INFO:tasks.workunit.client.0.smithi102.stdout: 9 16 447 431 0.187044 0.1875 0.324359 0.329529 2024-04-25T07:34:33.919 INFO:tasks.workunit.client.0.smithi102.stdout: 10 16 489 473 0.184744 0.164062 0.409438 0.330668 2024-04-25T07:34:34.919 INFO:tasks.workunit.client.0.smithi102.stdout: 11 16 543 527 0.187123 0.210938 0.30421 0.330755 2024-04-25T07:34:35.919 INFO:tasks.workunit.client.0.smithi102.stdout: 12 16 592 576 0.187477 0.191406 0.372588 0.330171 2024-04-25T07:34:36.919 INFO:tasks.workunit.client.0.smithi102.stdout: 13 16 641 625 0.187777 0.191406 0.339548 0.329568 2024-04-25T07:34:37.919 INFO:tasks.workunit.client.0.smithi102.stdout: 14 16 693 677 0.188872 0.203125 0.284132 0.327436 2024-04-25T07:34:38.920 INFO:tasks.workunit.client.0.smithi102.stdout: 15 16 738 722 0.187998 0.175781 0.378066 0.326068 2024-04-25T07:34:39.920 INFO:tasks.workunit.client.0.smithi102.stdout: 16 16 790 774 0.188941 0.203125 0.300274 0.326129 2024-04-25T07:34:40.920 INFO:tasks.workunit.client.0.smithi102.stdout: 17 16 849 833 0.191382 0.230469 0.321349 0.324965 2024-04-25T07:34:41.920 INFO:tasks.workunit.client.0.smithi102.stdout: 18 16 897 881 0.191165 0.1875 0.344768 0.324344 2024-04-25T07:34:42.920 INFO:tasks.workunit.client.0.smithi102.stdout: 19 16 947 931 0.191382 0.195312 0.325392 0.323607 2024-04-25T07:34:43.920 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:34:43.926265+0000 min lat: 0.212973 max lat: 0.831175 avg lat: 0.322409 2024-04-25T07:34:43.921 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:34:43.921 INFO:tasks.workunit.client.0.smithi102.stdout: 20 16 998 982 0.191773 0.199219 0.308607 0.322409 2024-04-25T07:34:44.921 INFO:tasks.workunit.client.0.smithi102.stdout: 21 16 1050 1034 0.192312 0.203125 0.265195 0.322549 2024-04-25T07:34:45.921 INFO:tasks.workunit.client.0.smithi102.stdout: 22 16 1103 1087 0.19298 0.207031 0.291949 0.321926 2024-04-25T07:34:46.921 INFO:tasks.workunit.client.0.smithi102.stdout: 23 16 1140 1124 0.190872 0.144531 0.467571 0.324437 2024-04-25T07:34:47.921 INFO:tasks.workunit.client.0.smithi102.stdout: 24 16 1193 1177 0.191544 0.207031 0.329882 0.323196 2024-04-25T07:34:48.921 INFO:tasks.workunit.client.0.smithi102.stdout: 25 16 1251 1235 0.192944 0.226562 0.314663 0.322687 2024-04-25T07:34:49.921 INFO:tasks.workunit.client.0.smithi102.stdout: 26 16 1305 1289 0.193635 0.210938 0.30053 0.321453 2024-04-25T07:34:50.921 INFO:tasks.workunit.client.0.smithi102.stdout: 27 16 1353 1337 0.193407 0.1875 0.282931 0.321175 2024-04-25T07:34:51.921 INFO:tasks.workunit.client.0.smithi102.stdout: 28 16 1409 1393 0.194311 0.21875 0.308299 0.320526 2024-04-25T07:34:52.922 INFO:tasks.workunit.client.0.smithi102.stdout: 29 16 1457 1441 0.194075 0.1875 0.33176 0.320504 2024-04-25T07:34:53.922 INFO:tasks.workunit.client.0.smithi102.stdout: 30 16 1506 1490 0.193985 0.191406 0.387295 0.320175 2024-04-25T07:34:54.922 INFO:tasks.workunit.client.0.smithi102.stdout:Total time run: 30.3488 2024-04-25T07:34:54.922 INFO:tasks.workunit.client.0.smithi102.stdout:Total writes made: 1506 2024-04-25T07:34:54.922 INFO:tasks.workunit.client.0.smithi102.stdout:Write size: 4096 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Object size: 4096 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Bandwidth (MB/sec): 0.19384 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Bandwidth: 0.0256526 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Max bandwidth (MB/sec): 0.230469 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Min bandwidth (MB/sec): 0.113281 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Average IOPS: 49 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev IOPS: 6.56707 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Max IOPS: 59 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Min IOPS: 29 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Average Latency(s): 0.320927 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Latency(s): 0.0529634 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Max latency(s): 0.831175 2024-04-25T07:34:54.923 INFO:tasks.workunit.client.0.smithi102.stdout:Min latency(s): 0.212973 2024-04-25T07:34:55.039 INFO:tasks.workunit.client.0.smithi102.stdout:after bench 2024-04-25T07:34:55.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-04-25T07:34:55.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 97314 2024-04-25T07:34:55.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-04-25T07:34:56.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 97314 2024-04-25T07:34:56.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (97314) - No such process 2024-04-25T07:34:56.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-04-25T07:34:56.631 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T07:34:56.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 99855 2024-04-25T07:34:56.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-04-25T07:34:57.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 99855 2024-04-25T07:34:57.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (99855) - No such process 2024-04-25T07:34:57.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-04-25T07:34:58.220 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already down. 2024-04-25T07:34:58.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 102421 2024-04-25T07:34:58.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-04-25T07:34:59.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 102421 2024-04-25T07:34:59.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (102421) - No such process 2024-04-25T07:34:59.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-04-25T07:34:59.807 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already down. 2024-04-25T07:34:59.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 104963 2024-04-25T07:34:59.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-04-25T07:35:00.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 104963 2024-04-25T07:35:00.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (104963) - No such process 2024-04-25T07:35:00.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-04-25T07:35:01.391 INFO:tasks.workunit.client.0.smithi102.stderr:osd.3 is already down. 2024-04-25T07:35:01.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:35:03.214 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:03.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:35:05.046 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:05.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:35:06.886 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:06.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:35:08.718 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:08.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-04-25T07:35:08.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-04-25T07:35:08.781 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:09.680 INFO:tasks.workunit.client.0.smithi102.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:35:09.680 INFO:tasks.workunit.client.0.smithi102.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:35:09.681 INFO:tasks.workunit.client.0.smithi102.stdout:2 : device size 0x100000000 : using 0x80495000(2.0 GiB) 2024-04-25T07:35:09.681 INFO:tasks.workunit.client.0.smithi102.stdout:Expanding DB/WAL... 2024-04-25T07:35:09.681 INFO:tasks.workunit.client.0.smithi102.stdout:0 : expanding to 0x536870912 2024-04-25T07:35:09.707 INFO:tasks.workunit.client.0.smithi102.stdout:0 : size label updated to 536870912 2024-04-25T07:35:09.708 INFO:tasks.workunit.client.0.smithi102.stdout:1 : expanding to 0x1073741824 2024-04-25T07:35:09.741 INFO:tasks.workunit.client.0.smithi102.stdout:1 : size label updated to 1073741824 2024-04-25T07:35:09.741 INFO:tasks.workunit.client.0.smithi102.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-04-25T07:35:10.001 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:35:10.003+0000 7f4d8524cac0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-04-25T07:35:10.001 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:35:10.003+0000 7f4d8524cac0 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-04-25T07:35:15.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-04-25T07:35:15.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-04-25T07:35:15.096 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:16.015 INFO:tasks.workunit.client.0.smithi102.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:35:16.016 INFO:tasks.workunit.client.0.smithi102.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:35:16.016 INFO:tasks.workunit.client.0.smithi102.stdout:2 : device size 0x101000000 : using 0x8149c000(2.0 GiB) 2024-04-25T07:35:16.016 INFO:tasks.workunit.client.0.smithi102.stdout:Expanding DB/WAL... 2024-04-25T07:35:16.016 INFO:tasks.workunit.client.0.smithi102.stdout:0 : expanding to 0x536870912 2024-04-25T07:35:16.047 INFO:tasks.workunit.client.0.smithi102.stdout:0 : size label updated to 536870912 2024-04-25T07:35:16.048 INFO:tasks.workunit.client.0.smithi102.stdout:1 : expanding to 0x1073741824 2024-04-25T07:35:16.073 INFO:tasks.workunit.client.0.smithi102.stdout:1 : size label updated to 1073741824 2024-04-25T07:35:16.073 INFO:tasks.workunit.client.0.smithi102.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-04-25T07:35:16.290 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:35:16.295+0000 7fa683d99ac0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-04-25T07:35:16.291 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:35:16.295+0000 7fa683d99ac0 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-04-25T07:35:21.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-04-25T07:35:21.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-04-25T07:35:21.664 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:22.582 INFO:tasks.workunit.client.0.smithi102.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:35:22.582 INFO:tasks.workunit.client.0.smithi102.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:35:22.582 INFO:tasks.workunit.client.0.smithi102.stdout:2 : device size 0x100020000 : using 0x804e2000(2.0 GiB) 2024-04-25T07:35:22.582 INFO:tasks.workunit.client.0.smithi102.stdout:Expanding DB/WAL... 2024-04-25T07:35:22.582 INFO:tasks.workunit.client.0.smithi102.stdout:0 : expanding to 0x536870912 2024-04-25T07:35:22.614 INFO:tasks.workunit.client.0.smithi102.stdout:0 : size label updated to 536870912 2024-04-25T07:35:22.614 INFO:tasks.workunit.client.0.smithi102.stdout:1 : expanding to 0x1073741824 2024-04-25T07:35:22.639 INFO:tasks.workunit.client.0.smithi102.stdout:1 : size label updated to 1073741824 2024-04-25T07:35:22.639 INFO:tasks.workunit.client.0.smithi102.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-04-25T07:35:22.865 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:35:22.867+0000 7effb6407ac0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-04-25T07:35:22.865 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:35:22.867+0000 7effb6407ac0 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-04-25T07:35:28.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-04-25T07:35:28.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-04-25T07:35:28.151 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:29.046 INFO:tasks.workunit.client.0.smithi102.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:35:29.047 INFO:tasks.workunit.client.0.smithi102.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:35:29.047 INFO:tasks.workunit.client.0.smithi102.stdout:2 : device size 0xfff00000 : using 0x803c0000(2.0 GiB) 2024-04-25T07:35:29.047 INFO:tasks.workunit.client.0.smithi102.stdout:Expanding DB/WAL... 2024-04-25T07:35:29.047 INFO:tasks.workunit.client.0.smithi102.stdout:0 : expanding to 0x536870912 2024-04-25T07:35:29.079 INFO:tasks.workunit.client.0.smithi102.stdout:0 : size label updated to 536870912 2024-04-25T07:35:29.079 INFO:tasks.workunit.client.0.smithi102.stdout:1 : expanding to 0x1073741824 2024-04-25T07:35:29.104 INFO:tasks.workunit.client.0.smithi102.stdout:1 : size label updated to 1073741824 2024-04-25T07:35:29.104 INFO:tasks.workunit.client.0.smithi102.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-04-25T07:35:29.330 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:35:29.331+0000 7f2b94952ac0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-04-25T07:35:29.330 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:35:29.331+0000 7f2b94952ac0 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-04-25T07:35:34.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:35:36.586 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:36.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:35:38.427 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:38.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:35:40.267 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:40.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:35:42.099 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-04-25T07:35:42.148 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:43.932 INFO:tasks.workunit.client.0.smithi102.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-04-25T07:35:43.933 INFO:tasks.workunit.client.0.smithi102.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-04-25T07:35:43.933 INFO:tasks.workunit.client.0.smithi102.stdout:2 : device size 0x100000000 : using 0x495000(4.6 MiB) 2024-04-25T07:35:43.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-04-25T07:35:43.961 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:45.743 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-04-25T07:35:45.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:35:47.063 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:47.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-04-25T07:35:47.109 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:48.891 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:35:48.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:35:50.207 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:50.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-04-25T07:35:50.255 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:52.287 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-04-25T07:35:52.287 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-04-25T07:35:52.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:35:53.115 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:53.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-04-25T07:35:53.153 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:53.153 INFO:tasks.workunit.client.0.smithi102.stderr:Migrate to WAL device isn't supported. 2024-04-25T07:35:53.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-04-25T07:35:53.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:35:54.967 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:54.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-04-25T07:35:55.015 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:57.047 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-04-25T07:35:57.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:35:58.367 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:35:58.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-04-25T07:35:58.414 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:59.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:36:01.019 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:36:01.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:36:01.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:36:01.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:36:01.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:36:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:36:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:36:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:36:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:36:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:36:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:36:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:36:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:36:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:36:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:36:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:36:01.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:36:01.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:36:01.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:36:01.043 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:36:01.043 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:01.044 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:36:01.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:36:01.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:36:01.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:36:01.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:36:01.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:36:01.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:36:01.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:36:01.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:36:01.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:36:01.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:36:01.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:36:01.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:36:01.046 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:36:01.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:36:01.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:36:01.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:36:01.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:36:01.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:36:01.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:36:01.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:36:01.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:01.078+0000 7f048168e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:01.081 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:01.082+0000 7f048168e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:01.083 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:01.086+0000 7f048168e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:01.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:36:01.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:36:01.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:36:01.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:36:01.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:36:01.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:01.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:36:01.463 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:36:01.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:01.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:01.877 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:01.878+0000 7f048168e7c0 -1 Falling back to public interface 2024-04-25T07:36:01.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:02.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:02.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:02.902 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:36:02.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:36:02.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:02.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:03.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:04.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:04.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:04.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:36:04.303 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:36:04.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:04.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:04.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:05.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:05.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:05.712 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:36:05.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:36:05.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:05.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:05.830 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:05.830+0000 7f048168e7c0 -1 osd.0 27 log_to_monitors true 2024-04-25T07:36:06.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:07.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:07.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:07.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:36:07.270 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:36:07.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:07.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:07.676 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/912843913,v1:127.0.0.1:6803/912843913] [v2:127.0.0.1:6804/912843913,v1:127.0.0.1:6805/912843913] exists,up 62b1a515-f1de-4a1b-865e-deaaf5cd7f26 2024-04-25T07:36:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:36:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:36:07.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:36:07.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=118425 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:36:07.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:36:07.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:36:07.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:36:07.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:36:07.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:36:07.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:36:07.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:36:07.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:36:07.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:36:07.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:36:07.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:36:07.683 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:36:07.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:36:07.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:36:07.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:36:07.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:36:07.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:36:07.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:36:07.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:36:07.716 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:07.718+0000 7f5d1b8667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:07.721 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:07.722+0000 7f5d1b8667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:07.722 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:07.722+0000 7f5d1b8667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:08.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:36:08.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:36:08.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:36:08.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:36:08.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:36:08.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:08.130 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:36:08.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:36:08.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:08.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:36:08.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:08.514+0000 7f5d1b8667c0 -1 Falling back to public interface 2024-04-25T07:36:08.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:09.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:09.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:09.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:36:09.572 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:36:09.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:09.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:36:10.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:11.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:11.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:11.022 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:36:11.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:36:11.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:11.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:36:11.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:12.295 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:12.298+0000 7f5d1b8667c0 -1 osd.1 30 log_to_monitors true 2024-04-25T07:36:12.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:12.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:12.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:36:12.483 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:36:12.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:12.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:36:13.060 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/3058731186,v1:127.0.0.1:6811/3058731186] [v2:127.0.0.1:6812/3058731186,v1:127.0.0.1:6813/3058731186] exists,up 51efadca-5b36-49e3-b603-35075dc6e371 2024-04-25T07:36:13.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:36:13.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:36:13.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:36:13.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=119389 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:36:13.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:36:13.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:36:13.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:36:13.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:36:13.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:36:13.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:36:13.065 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:36:13.065 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:13.065 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:36:13.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:36:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:36:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:36:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:36:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:36:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:36:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:36:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:36:13.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:36:13.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:36:13.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:36:13.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:36:13.067 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:36:13.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:36:13.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:36:13.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:36:13.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:36:13.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:36:13.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:36:13.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:36:13.102 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:13.101+0000 7f37098547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:13.202 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:13.201+0000 7f37098547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:13.203 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:13.205+0000 7f37098547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:13.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:36:13.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:36:13.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:36:13.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:36:13.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:36:13.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:13.716 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:36:13.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:36:13.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:13.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:36:13.756 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:13.757+0000 7f37098547c0 -1 Falling back to public interface 2024-04-25T07:36:14.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:15.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:15.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:15.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:36:15.338 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:36:15.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:15.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:36:15.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:16.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:16.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:16.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:36:16.741 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:36:16.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:16.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:36:16.860 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:16.861+0000 7f37098547c0 -1 osd.2 33 log_to_monitors true 2024-04-25T07:36:17.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:18.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:18.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:18.332 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:36:18.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:36:18.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:18.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:36:18.901 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 45 up_thru 45 down_at 34 last_clean_interval [15,33) [v2:127.0.0.1:6818/3150958577,v1:127.0.0.1:6819/3150958577] [v2:127.0.0.1:6820/3150958577,v1:127.0.0.1:6821/3150958577] exists,up 86c0edaf-9fed-4a03-a25d-5450ee8c066f 2024-04-25T07:36:18.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:36:18.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:36:18.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:36:18.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:36:18.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=120345 2024-04-25T07:36:18.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:36:18.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:36:18.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:36:18.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:36:18.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:36:18.906 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:36:18.906 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:18.906 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:36:18.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:36:18.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:36:18.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:36:18.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:36:18.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:36:18.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:36:18.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:36:18.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:36:18.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:36:18.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:36:18.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:36:18.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:36:18.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:36:18.908 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T07:36:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:36:18.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:36:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:36:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:36:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:36:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:36:18.939 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:18.941+0000 7f374d7aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:19.044 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:19.045+0000 7f374d7aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:19.046 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:19.045+0000 7f374d7aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:19.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:36:19.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:36:19.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:36:19.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:36:19.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:36:19.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:19.407 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:36:19.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:36:19.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:19.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:36:19.836 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:19.837+0000 7f374d7aa7c0 -1 Falling back to public interface 2024-04-25T07:36:19.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:20.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:20.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:20.852 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:36:20.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:36:20.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:20.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:36:21.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:22.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:22.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:22.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:36:22.264 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:36:22.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:22.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:36:22.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:23.703 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:36:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:36:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:36:23.949 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:36:23.949+0000 7f374d7aa7c0 -1 osd.3 35 log_to_monitors true 2024-04-25T07:36:24.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:25.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:25.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:25.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:36:25.152 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:36:25.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:25.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:36:26.029 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 49 up_thru 49 down_at 36 last_clean_interval [20,35) [v2:127.0.0.1:6826/1106927406,v1:127.0.0.1:6827/1106927406] [v2:127.0.0.1:6828/1106927406,v1:127.0.0.1:6829/1106927406] exists,up 8271f3ee-32c9-4e2f-a4cb-99671ce33ed0 2024-04-25T07:36:26.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:36:26.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:36:26.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:36:26.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:36:26.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=121279 2024-04-25T07:36:26.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-04-25T07:36:26.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:36:26.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:36:26.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:36:26.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:36:26.033 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:36:26.033 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:36:26.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:36:26.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:36:26.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:36:26.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:36:26.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:36:26.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:36:26.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:36:26.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:36:26.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:36:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:36:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:36:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T07:36:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T07:36:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:36:26.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:36:26.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:36:26.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757255 2024-04-25T07:36:26.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757255 2024-04-25T07:36:26.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-163208757255' 2024-04-25T07:36:26.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:36:26.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:36:27.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=176093659141 2024-04-25T07:36:27.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 176093659141 2024-04-25T07:36:27.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-163208757255 1-176093659141' 2024-04-25T07:36:27.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:36:27.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:36:27.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528325 2024-04-25T07:36:27.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528325 2024-04-25T07:36:27.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528325' 2024-04-25T07:36:27.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:36:27.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:36:27.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=210453397507 2024-04-25T07:36:27.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 210453397507 2024-04-25T07:36:27.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528325 3-210453397507' 2024-04-25T07:36:27.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:36:27.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:36:27.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-163208757255 2024-04-25T07:36:27.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:36:27.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-163208757255 2024-04-25T07:36:27.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:36:27.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757255 2024-04-25T07:36:27.376 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 163208757255 2024-04-25T07:36:27.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 163208757255' 2024-04-25T07:36:27.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:36:27.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757252 -lt 163208757255 2024-04-25T07:36:27.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:36:28.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:36:28.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:36:29.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757255 -lt 163208757255 2024-04-25T07:36:29.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:36:29.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-176093659141 2024-04-25T07:36:29.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:36:29.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:36:29.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-176093659141 2024-04-25T07:36:29.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:36:29.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=176093659141 2024-04-25T07:36:29.252 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 176093659141 2024-04-25T07:36:29.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 176093659141' 2024-04-25T07:36:29.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:36:29.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 176093659142 -lt 176093659141 2024-04-25T07:36:29.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:36:29.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:36:29.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-193273528325 2024-04-25T07:36:29.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:36:29.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-193273528325 2024-04-25T07:36:29.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:36:29.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528325 2024-04-25T07:36:29.736 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 193273528325 2024-04-25T07:36:29.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 193273528325' 2024-04-25T07:36:29.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:36:30.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528325 -lt 193273528325 2024-04-25T07:36:30.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:36:30.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-210453397507 2024-04-25T07:36:30.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:36:30.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:36:30.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-210453397507 2024-04-25T07:36:30.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:36:30.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=210453397507 2024-04-25T07:36:30.185 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 210453397507 2024-04-25T07:36:30.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 210453397507' 2024-04-25T07:36:30.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:36:30.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 210453397508 -lt 210453397507 2024-04-25T07:36:30.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:36:30.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:36:30.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:36:31.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:36:31.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:36:31.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:36:31.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:36:31.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:36:31.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:36:31.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:36:31.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:36:31.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:36:31.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:36:31.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:36:31.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:36:32.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:36:32.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:36:32.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:36:32.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:36:32.296 INFO:tasks.workunit.client.0.smithi102.stdout:hints = 1 2024-04-25T07:36:32.296 INFO:tasks.workunit.client.0.smithi102.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:36:32.296 INFO:tasks.workunit.client.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_122596 2024-04-25T07:36:32.296 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:36:32.296 INFO:tasks.workunit.client.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:36:33.296 INFO:tasks.workunit.client.0.smithi102.stdout: 1 16 61 45 0.175764 0.175781 0.181457 0.289602 2024-04-25T07:36:34.296 INFO:tasks.workunit.client.0.smithi102.stdout: 2 16 116 100 0.195289 0.214844 0.216777 0.290897 2024-04-25T07:36:35.296 INFO:tasks.workunit.client.0.smithi102.stdout: 3 16 171 155 0.201797 0.214844 0.235262 0.291342 2024-04-25T07:36:36.296 INFO:tasks.workunit.client.0.smithi102.stdout: 4 16 223 207 0.202124 0.203125 0.360397 0.296947 2024-04-25T07:36:37.296 INFO:tasks.workunit.client.0.smithi102.stdout: 5 16 270 254 0.198414 0.183594 0.327465 0.306327 2024-04-25T07:36:38.296 INFO:tasks.workunit.client.0.smithi102.stdout: 6 16 320 304 0.197894 0.195312 0.431143 0.307263 2024-04-25T07:36:39.297 INFO:tasks.workunit.client.0.smithi102.stdout: 7 16 370 354 0.197514 0.195312 0.20852 0.306822 2024-04-25T07:36:40.297 INFO:tasks.workunit.client.0.smithi102.stdout: 8 16 422 406 0.198212 0.203125 0.25394 0.309054 2024-04-25T07:36:41.297 INFO:tasks.workunit.client.0.smithi102.stdout: 9 16 469 453 0.196585 0.183594 0.267439 0.310506 2024-04-25T07:36:42.297 INFO:tasks.workunit.client.0.smithi102.stdout: 10 16 520 504 0.196846 0.199219 0.262442 0.314438 2024-04-25T07:36:43.297 INFO:tasks.workunit.client.0.smithi102.stdout: 11 16 569 553 0.196349 0.191406 0.250603 0.312288 2024-04-25T07:36:44.297 INFO:tasks.workunit.client.0.smithi102.stdout: 12 16 622 606 0.197238 0.207031 0.26803 0.313499 2024-04-25T07:36:45.297 INFO:tasks.workunit.client.0.smithi102.stdout: 13 16 666 650 0.195285 0.171875 0.258858 0.314517 2024-04-25T07:36:46.298 INFO:tasks.workunit.client.0.smithi102.stdout: 14 16 722 706 0.196958 0.21875 0.230916 0.314054 2024-04-25T07:36:47.298 INFO:tasks.workunit.client.0.smithi102.stdout: 15 16 768 752 0.195805 0.179688 0.264833 0.315939 2024-04-25T07:36:48.298 INFO:tasks.workunit.client.0.smithi102.stdout: 16 16 821 805 0.196505 0.207031 0.282141 0.315282 2024-04-25T07:36:49.298 INFO:tasks.workunit.client.0.smithi102.stdout: 17 16 869 853 0.195974 0.1875 0.160336 0.316574 2024-04-25T07:36:50.298 INFO:tasks.workunit.client.0.smithi102.stdout: 18 16 927 911 0.197672 0.226562 0.259503 0.314237 2024-04-25T07:36:51.298 INFO:tasks.workunit.client.0.smithi102.stdout: 19 16 974 958 0.196929 0.183594 0.212655 0.315135 2024-04-25T07:36:52.298 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:36:52.303758+0000 min lat: 0.129983 max lat: 0.501714 avg lat: 0.313275 2024-04-25T07:36:52.299 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:36:52.299 INFO:tasks.workunit.client.0.smithi102.stdout: 20 16 1032 1016 0.198409 0.226562 0.368614 0.313275 2024-04-25T07:36:53.299 INFO:tasks.workunit.client.0.smithi102.stdout: 21 16 1099 1083 0.201422 0.261719 0.159003 0.308285 2024-04-25T07:36:54.299 INFO:tasks.workunit.client.0.smithi102.stdout: 22 15 1163 1148 0.203806 0.253906 0.283302 0.305009 2024-04-25T07:36:55.299 INFO:tasks.workunit.client.0.smithi102.stdout: 23 16 1230 1214 0.206153 0.257812 0.273857 0.301822 2024-04-25T07:36:56.299 INFO:tasks.workunit.client.0.smithi102.stdout: 24 16 1286 1270 0.206677 0.21875 0.329522 0.300191 2024-04-25T07:36:57.299 INFO:tasks.workunit.client.0.smithi102.stdout: 25 16 1340 1324 0.206846 0.210938 0.277457 0.300178 2024-04-25T07:36:58.299 INFO:tasks.workunit.client.0.smithi102.stdout: 26 16 1397 1381 0.207453 0.222656 0.310287 0.299144 2024-04-25T07:36:59.300 INFO:tasks.workunit.client.0.smithi102.stdout: 27 16 1457 1441 0.208449 0.234375 0.308092 0.298405 2024-04-25T07:37:00.300 INFO:tasks.workunit.client.0.smithi102.stdout: 28 16 1526 1510 0.210629 0.269531 0.204927 0.296164 2024-04-25T07:37:01.300 INFO:tasks.workunit.client.0.smithi102.stdout: 29 16 1581 1565 0.210774 0.214844 0.3167 0.294719 2024-04-25T07:37:02.300 INFO:tasks.workunit.client.0.smithi102.stdout: 30 16 1645 1629 0.212081 0.25 0.233223 0.29299 2024-04-25T07:37:03.300 INFO:tasks.workunit.client.0.smithi102.stdout:Total time run: 30.2718 2024-04-25T07:37:03.300 INFO:tasks.workunit.client.0.smithi102.stdout:Total writes made: 1645 2024-04-25T07:37:03.300 INFO:tasks.workunit.client.0.smithi102.stdout:Write size: 4096 2024-04-25T07:37:03.300 INFO:tasks.workunit.client.0.smithi102.stdout:Object size: 4096 2024-04-25T07:37:03.300 INFO:tasks.workunit.client.0.smithi102.stdout:Bandwidth (MB/sec): 0.21227 2024-04-25T07:37:03.301 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Bandwidth: 0.0266351 2024-04-25T07:37:03.301 INFO:tasks.workunit.client.0.smithi102.stdout:Max bandwidth (MB/sec): 0.269531 2024-04-25T07:37:03.301 INFO:tasks.workunit.client.0.smithi102.stdout:Min bandwidth (MB/sec): 0.171875 2024-04-25T07:37:03.301 INFO:tasks.workunit.client.0.smithi102.stdout:Average IOPS: 54 2024-04-25T07:37:03.301 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev IOPS: 6.81859 2024-04-25T07:37:03.301 INFO:tasks.workunit.client.0.smithi102.stdout:Max IOPS: 69 2024-04-25T07:37:03.301 INFO:tasks.workunit.client.0.smithi102.stdout:Min IOPS: 44 2024-04-25T07:37:03.301 INFO:tasks.workunit.client.0.smithi102.stdout:Average Latency(s): 0.293735 2024-04-25T07:37:03.301 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Latency(s): 0.0643843 2024-04-25T07:37:03.301 INFO:tasks.workunit.client.0.smithi102.stdout:Max latency(s): 0.501714 2024-04-25T07:37:03.301 INFO:tasks.workunit.client.0.smithi102.stdout:Min latency(s): 0.129983 2024-04-25T07:37:03.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 118425 2024-04-25T07:37:03.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-04-25T07:37:04.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 118425 2024-04-25T07:37:04.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (118425) - No such process 2024-04-25T07:37:04.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-04-25T07:37:05.003 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T07:37:05.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 119389 2024-04-25T07:37:05.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-04-25T07:37:06.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 119389 2024-04-25T07:37:06.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (119389) - No such process 2024-04-25T07:37:06.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-04-25T07:37:06.541 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already down. 2024-04-25T07:37:06.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 120345 2024-04-25T07:37:06.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-04-25T07:37:07.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 120345 2024-04-25T07:37:07.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (120345) - No such process 2024-04-25T07:37:07.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-04-25T07:37:07.985 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already down. 2024-04-25T07:37:08.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 121279 2024-04-25T07:37:08.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-04-25T07:37:09.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 121279 2024-04-25T07:37:09.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (121279) - No such process 2024-04-25T07:37:09.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-04-25T07:37:09.573 INFO:tasks.workunit.client.0.smithi102.stderr:osd.3 is already down. 2024-04-25T07:37:09.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:37:10.895 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:37:10.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-04-25T07:37:11.203 INFO:tasks.workunit.client.0.smithi102.stderr:512+0 records in 2024-04-25T07:37:11.203 INFO:tasks.workunit.client.0.smithi102.stderr:512+0 records out 2024-04-25T07:37:11.203 INFO:tasks.workunit.client.0.smithi102.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.284028 s, 1.9 GB/s 2024-04-25T07:37:11.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-04-25T07:37:11.225 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:37:16.263 INFO:tasks.workunit.client.0.smithi102.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-04-25T07:37:16.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:37:18.079 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:37:18.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:37:19.407 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:37:19.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-04-25T07:37:19.983 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records in 2024-04-25T07:37:19.984 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records out 2024-04-25T07:37:19.984 INFO:tasks.workunit.client.0.smithi102.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.563896 s, 1.9 GB/s 2024-04-25T07:37:19.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-04-25T07:37:20.007 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:37:28.043 INFO:tasks.workunit.client.0.smithi102.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-04-25T07:37:28.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-04-25T07:37:28.074 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:37:30.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:37:31.939 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:37:31.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:37:32.771 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:37:32.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-04-25T07:37:32.807 INFO:tasks.workunit.client.0.smithi102.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:37:32.807 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:37:33.587 INFO:tasks.workunit.client.0.smithi102.stdout:DB device added 2024-04-25T07:37:33.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-04-25T07:37:33.619 INFO:tasks.workunit.client.0.smithi102.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:37:33.620 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:37:34.903 INFO:tasks.workunit.client.0.smithi102.stdout:WAL device added 2024-04-25T07:37:34.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-04-25T07:37:34.934 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:37:36.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:37:38.787 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:37:38.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:37:40.131 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:37:40.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-04-25T07:37:40.173 INFO:tasks.workunit.client.0.smithi102.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:37:40.173 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:37:41.467 INFO:tasks.workunit.client.0.smithi102.stdout:WAL device added 2024-04-25T07:37:41.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-04-25T07:37:41.499 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:37:43.783 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-04-25T07:37:43.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:37:45.103 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:37:45.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:37:45.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:37:45.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:37:45.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:37:45.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:37:45.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:37:45.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:37:45.124 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:37:45.124 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:37:45.124 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:37:45.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:37:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:37:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:37:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:37:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:37:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:37:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:37:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:37:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:37:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:37:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:37:45.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:37:45.126 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:37:45.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:37:45.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:37:45.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:37:45.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:37:45.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:37:45.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:37:45.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:45.167+0000 7f7565bc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:45.166 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:45.167+0000 7f7565bc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:45.168 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:45.171+0000 7f7565bc57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:37:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:37:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:37:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:37:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:37:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:45.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:37:45.539 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:37:45.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:45.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:46.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:46.202 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:46.203+0000 7f7565bc57c0 -1 Falling back to public interface 2024-04-25T07:37:47.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:47.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:47.017 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:37:47.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:37:47.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:47.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:47.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:48.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:48.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:48.439 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:37:48.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:37:48.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:48.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:48.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:49.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:49.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:49.850 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:37:49.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:37:49.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:49.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:50.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:51.198 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:51.199+0000 7f7565bc57c0 -1 osd.0 51 log_to_monitors true 2024-04-25T07:37:51.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:51.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:51.256 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:37:51.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:37:51.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:51.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:37:51.786 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 62 up_thru 49 down_at 52 last_clean_interval [38,51) [v2:127.0.0.1:6802/741414063,v1:127.0.0.1:6803/741414063] [v2:127.0.0.1:6804/741414063,v1:127.0.0.1:6805/741414063] exists,up 62b1a515-f1de-4a1b-865e-deaaf5cd7f26 2024-04-25T07:37:51.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:37:51.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:37:51.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:37:51.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=128704 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:37:51.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:37:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:37:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:37:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:37:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:37:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:37:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:37:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:37:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:37:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:37:51.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:37:51.794 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:37:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:37:51.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:37:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:37:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:37:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:37:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:37:51.818 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:51.819+0000 7f7e3b7df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:51.876 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:51.879+0000 7f7e3b7df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:51.878 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:51.879+0000 7f7e3b7df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:52.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:37:52.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:37:52.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:37:52.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:37:52.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:37:52.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:52.277 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:37:52.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:37:52.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:52.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:52.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:52.926 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:52.927+0000 7f7e3b7df7c0 -1 Falling back to public interface 2024-04-25T07:37:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:53.722 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:37:53.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:37:53.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:53.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:54.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:55.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:55.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:55.220 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:37:55.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:37:55.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:55.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:55.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:56.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:56.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:56.651 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:37:56.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:37:56.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:56.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:57.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:57.708 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:57.711+0000 7f7e3b7df7c0 -1 osd.1 54 log_to_monitors true 2024-04-25T07:37:58.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:58.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:58.052 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:37:58.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:37:58.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:58.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:37:58.756 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 55 last_clean_interval [41,54) [v2:127.0.0.1:6810/3942435030,v1:127.0.0.1:6811/3942435030] [v2:127.0.0.1:6812/3942435030,v1:127.0.0.1:6813/3942435030] exists,up 51efadca-5b36-49e3-b603-35075dc6e371 2024-04-25T07:37:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:37:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:37:58.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:37:58.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:37:58.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=129674 2024-04-25T07:37:58.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:37:58.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:37:58.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:37:58.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:37:58.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:37:58.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:37:58.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:37:58.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:37:58.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:37:58.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:37:58.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:37:58.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:37:58.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:37:58.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:37:58.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:37:58.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:37:58.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:37:58.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:37:58.762 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:37:58.762 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:37:58.762 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:37:58.763 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:37:58.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:37:58.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:37:58.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:37:58.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:37:58.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:37:58.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:37:58.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:37:58.792 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:58.795+0000 7f4e6fec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:58.794 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:58.795+0000 7f4e6fec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:58.796 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:58.799+0000 7f4e6fec47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:59.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:37:59.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:37:59.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:37:59.227 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:37:59.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:37:59.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:37:59.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:59.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:37:59.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:37:59.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:59.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:59.834 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:37:59.835+0000 7f4e6fec47c0 -1 Falling back to public interface 2024-04-25T07:38:00.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:00.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:00.698 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:38:00.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:38:00.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:00.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:01.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:02.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:02.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:02.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:38:02.116 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:38:02.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:02.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:02.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:03.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:03.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:03.534 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:38:03.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:38:03.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:03.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:03.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:04.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:04.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:04.955 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:38:04.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:38:04.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:04.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:05.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:05.413 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:38:05.415+0000 7f4e6fec47c0 -1 osd.2 57 log_to_monitors true 2024-04-25T07:38:06.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:06.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:06.370 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:38:06.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:38:06.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:06.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:07.031 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 68 up_thru 55 down_at 58 last_clean_interval [45,57) [v2:127.0.0.1:6818/2899345656,v1:127.0.0.1:6819/2899345656] [v2:127.0.0.1:6820/2899345656,v1:127.0.0.1:6821/2899345656] exists,up 86c0edaf-9fed-4a03-a25d-5450ee8c066f 2024-04-25T07:38:07.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:38:07.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:38:07.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:38:07.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:38:07.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=130643 2024-04-25T07:38:07.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:38:07.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:07.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:38:07.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:38:07.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:38:07.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:38:07.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:38:07.038 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T07:38:07.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:38:07.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:38:07.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:38:07.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:38:07.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:38:07.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:38:07.063 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:38:07.067+0000 7f93608ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:07.066 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:38:07.067+0000 7f93608ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:07.068 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:38:07.071+0000 7f93608ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:07.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:38:07.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:38:07.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:38:07.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:38:07.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:38:07.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:07.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:38:07.810 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:38:07.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:07.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:38:07.885 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:38:07.887+0000 7f93608ce7c0 -1 Falling back to public interface 2024-04-25T07:38:08.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:09.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:09.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:09.265 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:38:09.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:38:09.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:09.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:38:09.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:10.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:10.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:10.685 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:38:10.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:38:10.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:10.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:38:11.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:12.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:12.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:12.086 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:38:12.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:38:12.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:12.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:38:12.481 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:38:12.483+0000 7f93608ce7c0 -1 osd.3 59 log_to_monitors true 2024-04-25T07:38:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:13.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:13.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:13.657 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:38:13.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:38:13.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:13.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:38:14.111 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 73 up_thru 58 down_at 60 last_clean_interval [49,59) [v2:127.0.0.1:6826/473773476,v1:127.0.0.1:6827/473773476] [v2:127.0.0.1:6828/473773476,v1:127.0.0.1:6829/473773476] exists,up 8271f3ee-32c9-4e2f-a4cb-99671ce33ed0 2024-04-25T07:38:14.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:38:14.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:38:14.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:38:14.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:38:14.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=131637 2024-04-25T07:38:14.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:38:14.300 INFO:tasks.workunit.client.0.smithi102.stdout:hints = 1 2024-04-25T07:38:14.300 INFO:tasks.workunit.client.0.smithi102.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:38:14.300 INFO:tasks.workunit.client.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_132586 2024-04-25T07:38:14.300 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:38:14.300 INFO:tasks.workunit.client.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:38:15.300 INFO:tasks.workunit.client.0.smithi102.stdout: 1 16 23 7 0.0273421 0.0273438 0.545956 0.446319 2024-04-25T07:38:16.302 INFO:tasks.workunit.client.0.smithi102.stdout: 2 16 58 42 0.0820208 0.136719 0.358843 0.657498 2024-04-25T07:38:17.300 INFO:tasks.workunit.client.0.smithi102.stdout: 3 16 106 90 0.117173 0.1875 0.34432 0.50394 2024-04-25T07:38:18.301 INFO:tasks.workunit.client.0.smithi102.stdout: 4 16 143 127 0.124008 0.144531 0.253265 0.469052 2024-04-25T07:38:19.300 INFO:tasks.workunit.client.0.smithi102.stdout: 5 16 198 182 0.14217 0.214844 0.386246 0.428403 2024-04-25T07:38:20.301 INFO:tasks.workunit.client.0.smithi102.stdout: 6 16 231 215 0.139957 0.128906 0.371428 0.423258 2024-04-25T07:38:21.301 INFO:tasks.workunit.client.0.smithi102.stdout: 7 16 281 265 0.147861 0.195312 0.327096 0.411871 2024-04-25T07:38:22.301 INFO:tasks.workunit.client.0.smithi102.stdout: 8 16 322 306 0.149396 0.160156 0.402839 0.4057 2024-04-25T07:38:23.301 INFO:tasks.workunit.client.0.smithi102.stdout: 9 16 370 354 0.153627 0.1875 0.407714 0.401611 2024-04-25T07:38:24.301 INFO:tasks.workunit.client.0.smithi102.stdout: 10 16 411 395 0.154278 0.160156 0.383746 0.398276 2024-04-25T07:38:25.301 INFO:tasks.workunit.client.0.smithi102.stdout: 11 16 456 440 0.15623 0.175781 0.474843 0.396157 2024-04-25T07:38:26.301 INFO:tasks.workunit.client.0.smithi102.stdout: 12 16 492 476 0.154928 0.140625 0.465125 0.396474 2024-04-25T07:38:27.302 INFO:tasks.workunit.client.0.smithi102.stdout: 13 16 541 525 0.157732 0.191406 0.349168 0.391228 2024-04-25T07:38:28.302 INFO:tasks.workunit.client.0.smithi102.stdout: 14 16 582 566 0.157904 0.160156 0.410113 0.389435 2024-04-25T07:38:29.302 INFO:tasks.workunit.client.0.smithi102.stdout: 15 16 637 621 0.161698 0.214844 0.203817 0.384576 2024-04-25T07:38:30.302 INFO:tasks.workunit.client.0.smithi102.stdout: 16 16 673 657 0.16038 0.140625 0.477246 0.383775 2024-04-25T07:38:31.302 INFO:tasks.workunit.client.0.smithi102.stdout: 17 16 719 703 0.161514 0.179688 0.2047 0.382631 2024-04-25T07:38:32.302 INFO:tasks.workunit.client.0.smithi102.stdout: 18 16 771 755 0.163824 0.203125 0.246192 0.378865 2024-04-25T07:38:33.303 INFO:tasks.workunit.client.0.smithi102.stdout: 19 16 815 799 0.164247 0.171875 0.367788 0.37768 2024-04-25T07:38:34.303 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:38:34.307506+0000 min lat: 0.197312 max lat: 1.33915 avg lat: 0.375966 2024-04-25T07:38:34.303 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:38:34.303 INFO:tasks.workunit.client.0.smithi102.stdout: 20 16 857 841 0.164237 0.164062 0.367769 0.375966 2024-04-25T07:38:35.303 INFO:tasks.workunit.client.0.smithi102.stdout: 21 16 900 884 0.164413 0.167969 0.39206 0.374993 2024-04-25T07:38:36.303 INFO:tasks.workunit.client.0.smithi102.stdout: 22 16 948 932 0.165462 0.1875 0.338412 0.375038 2024-04-25T07:38:37.303 INFO:tasks.workunit.client.0.smithi102.stdout: 23 16 990 974 0.1654 0.164062 0.333654 0.374391 2024-04-25T07:38:38.303 INFO:tasks.workunit.client.0.smithi102.stdout: 24 16 1048 1032 0.167947 0.226562 0.204307 0.371378 2024-04-25T07:38:39.303 INFO:tasks.workunit.client.0.smithi102.stdout: 25 16 1085 1069 0.16701 0.144531 0.37771 0.370239 2024-04-25T07:38:40.303 INFO:tasks.workunit.client.0.smithi102.stdout: 26 16 1137 1121 0.168398 0.203125 0.201124 0.368813 2024-04-25T07:38:41.303 INFO:tasks.workunit.client.0.smithi102.stdout: 27 16 1174 1158 0.167513 0.144531 0.377095 0.370279 2024-04-25T07:38:42.304 INFO:tasks.workunit.client.0.smithi102.stdout: 28 16 1226 1210 0.168784 0.203125 0.373918 0.368749 2024-04-25T07:38:43.303 INFO:tasks.workunit.client.0.smithi102.stdout: 29 16 1266 1250 0.168351 0.15625 0.247827 0.369096 2024-04-25T07:38:44.304 INFO:tasks.workunit.client.0.smithi102.stdout: 30 16 1314 1298 0.168989 0.1875 0.353764 0.368167 2024-04-25T07:38:45.304 INFO:tasks.workunit.client.0.smithi102.stdout:Total time run: 30.2645 2024-04-25T07:38:45.304 INFO:tasks.workunit.client.0.smithi102.stdout:Total writes made: 1314 2024-04-25T07:38:45.304 INFO:tasks.workunit.client.0.smithi102.stdout:Write size: 4096 2024-04-25T07:38:45.304 INFO:tasks.workunit.client.0.smithi102.stdout:Object size: 4096 2024-04-25T07:38:45.304 INFO:tasks.workunit.client.0.smithi102.stdout:Bandwidth (MB/sec): 0.169598 2024-04-25T07:38:45.304 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Bandwidth: 0.0372835 2024-04-25T07:38:45.305 INFO:tasks.workunit.client.0.smithi102.stdout:Max bandwidth (MB/sec): 0.226562 2024-04-25T07:38:45.305 INFO:tasks.workunit.client.0.smithi102.stdout:Min bandwidth (MB/sec): 0.0273438 2024-04-25T07:38:45.305 INFO:tasks.workunit.client.0.smithi102.stdout:Average IOPS: 43 2024-04-25T07:38:45.305 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev IOPS: 9.54457 2024-04-25T07:38:45.305 INFO:tasks.workunit.client.0.smithi102.stdout:Max IOPS: 58 2024-04-25T07:38:45.305 INFO:tasks.workunit.client.0.smithi102.stdout:Min IOPS: 7 2024-04-25T07:38:45.305 INFO:tasks.workunit.client.0.smithi102.stdout:Average Latency(s): 0.367493 2024-04-25T07:38:45.305 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Latency(s): 0.108116 2024-04-25T07:38:45.305 INFO:tasks.workunit.client.0.smithi102.stdout:Max latency(s): 1.33915 2024-04-25T07:38:45.305 INFO:tasks.workunit.client.0.smithi102.stdout:Min latency(s): 0.197312 2024-04-25T07:38:45.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 128704 2024-04-25T07:38:45.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-04-25T07:38:46.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 128704 2024-04-25T07:38:46.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (128704) - No such process 2024-04-25T07:38:46.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-04-25T07:38:47.030 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T07:38:47.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 129674 2024-04-25T07:38:47.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-04-25T07:38:48.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 129674 2024-04-25T07:38:48.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (129674) - No such process 2024-04-25T07:38:48.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-04-25T07:38:48.559 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already down. 2024-04-25T07:38:48.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 130643 2024-04-25T07:38:48.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-04-25T07:38:49.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 130643 2024-04-25T07:38:49.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (130643) - No such process 2024-04-25T07:38:49.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-04-25T07:38:50.011 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already down. 2024-04-25T07:38:50.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 131637 2024-04-25T07:38:50.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-04-25T07:38:51.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 131637 2024-04-25T07:38:51.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (131637) - No such process 2024-04-25T07:38:51.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-04-25T07:38:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:osd.3 is already down. 2024-04-25T07:38:51.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:38:53.415 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:38:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-04-25T07:38:54.000 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records in 2024-04-25T07:38:54.000 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records out 2024-04-25T07:38:54.000 INFO:tasks.workunit.client.0.smithi102.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.562256 s, 1.9 GB/s 2024-04-25T07:38:54.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-04-25T07:38:54.023 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:39:03.059 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-04-25T07:39:03.059 INFO:tasks.workunit.client.0.smithi102.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-04-25T07:39:03.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:39:04.875 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:39:04.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-04-25T07:39:05.195 INFO:tasks.workunit.client.0.smithi102.stderr:512+0 records in 2024-04-25T07:39:05.195 INFO:tasks.workunit.client.0.smithi102.stderr:512+0 records out 2024-04-25T07:39:05.196 INFO:tasks.workunit.client.0.smithi102.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.299858 s, 1.8 GB/s 2024-04-25T07:39:05.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-04-25T07:39:05.220 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:39:10.751 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-04-25T07:39:10.751 INFO:tasks.workunit.client.0.smithi102.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-04-25T07:39:10.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-04-25T07:39:10.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:39:12.647 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:39:12.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:39:14.483 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:39:14.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-04-25T07:39:15.069 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records in 2024-04-25T07:39:15.069 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records out 2024-04-25T07:39:15.069 INFO:tasks.workunit.client.0.smithi102.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.561768 s, 1.9 GB/s 2024-04-25T07:39:15.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-04-25T07:39:15.092 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:39:24.127 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:39:24.128 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-04-25T07:39:24.128 INFO:tasks.workunit.client.0.smithi102.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-04-25T07:39:24.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-04-25T07:39:24.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:39:25.563 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:39:25.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-04-25T07:39:25.606 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:39:27.387 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:39:27.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-04-25T07:39:27.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:39:28.323 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:39:28.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-04-25T07:39:30.147 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:39:30.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-04-25T07:39:30.740 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records in 2024-04-25T07:39:30.740 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records out 2024-04-25T07:39:30.740 INFO:tasks.workunit.client.0.smithi102.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.566385 s, 1.9 GB/s 2024-04-25T07:39:30.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-04-25T07:39:30.763 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:39:30.764 INFO:tasks.workunit.client.0.smithi102.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-04-25T07:39:30.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-04-25T07:39:30.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:39:32.575 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:39:32.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-04-25T07:39:39.876 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records in 2024-04-25T07:39:39.876 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records out 2024-04-25T07:39:39.876 INFO:tasks.workunit.client.0.smithi102.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.14346 s, 150 MB/s 2024-04-25T07:39:39.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-04-25T07:39:39.903 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:39:42.183 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-04-25T07:39:42.183 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-04-25T07:39:42.183 INFO:tasks.workunit.client.0.smithi102.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-04-25T07:39:42.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:39:43.503 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:39:43.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-04-25T07:39:44.087 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records in 2024-04-25T07:39:44.087 INFO:tasks.workunit.client.0.smithi102.stderr:1024+0 records out 2024-04-25T07:39:44.087 INFO:tasks.workunit.client.0.smithi102.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.566074 s, 1.9 GB/s 2024-04-25T07:39:44.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-04-25T07:39:44.104 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:39:52.387 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-04-25T07:39:52.388 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-04-25T07:39:52.388 INFO:tasks.workunit.client.0.smithi102.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-04-25T07:39:52.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:39:53.703 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:39:53.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:39:53.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:39:53.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:39:53.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:39:53.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:39:53.724 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:39:53.724 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:39:53.724 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:39:53.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:39:53.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:39:53.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:39:53.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:39:53.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:39:53.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:39:53.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:39:53.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:39:53.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:39:53.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:39:53.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:39:53.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:39:53.725 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:39:53.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:39:53.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:39:53.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:39:53.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:39:53.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:39:53.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:39:53.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:39:53.758 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:39:53.757+0000 7fb5c8f837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:53.760 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:39:53.761+0000 7fb5c8f837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:53.762 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:39:53.761+0000 7fb5c8f837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:39:54.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:39:54.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:39:54.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:39:54.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:39:54.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:39:54.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:54.159 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:39:54.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:39:54.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:54.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:54.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:54.798 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:39:54.797+0000 7fb5c8f837c0 -1 Falling back to public interface 2024-04-25T07:39:55.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:55.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:55.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:39:55.591 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:39:55.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:55.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:56.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:57.010 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:39:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:39:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:57.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:57.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:58.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:58.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:58.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:39:58.432 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:39:58.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:58.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:39:58.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:39:59.371 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:39:59.372+0000 7fb5c8f837c0 -1 osd.0 75 log_to_monitors true 2024-04-25T07:39:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:39:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:39:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:39:59.843 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:39:59.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:39:59.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:00.337 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 86 up_thru 86 down_at 76 last_clean_interval [62,75) [v2:127.0.0.1:6802/412629632,v1:127.0.0.1:6803/412629632] [v2:127.0.0.1:6804/412629632,v1:127.0.0.1:6805/412629632] exists,up 62b1a515-f1de-4a1b-865e-deaaf5cd7f26 2024-04-25T07:40:00.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:40:00.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:40:00.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:40:00.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=138716 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:40:00.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:40:00.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:40:00.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:40:00.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:40:00.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:40:00.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:40:00.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:00.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:40:00.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:40:00.345 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:40:00.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:40:00.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:40:00.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:40:00.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:40:00.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:40:00.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:40:00.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:40:00.373 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:00.372+0000 7f13bc8d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:00.376 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:00.376+0000 7f13bc8d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:00.378 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:00.376+0000 7f13bc8d27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:00.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:40:00.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:40:00.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:40:00.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:40:00.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:40:00.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:00.810 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:40:00.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:40:00.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:00.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:00.940 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:00.940+0000 7f13bc8d27c0 -1 Falling back to public interface 2024-04-25T07:40:01.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:02.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:02.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:02.237 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:40:02.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:40:02.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:02.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:02.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:03.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:03.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:03.646 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:40:03.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:40:03.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:03.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:04.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:04.302 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:04.300+0000 7f13bc8d27c0 -1 osd.1 78 log_to_monitors true 2024-04-25T07:40:05.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:05.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:05.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:40:05.117 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:40:05.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:05.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:05.691 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 89 up_thru 89 down_at 79 last_clean_interval [65,78) [v2:127.0.0.1:6810/386250626,v1:127.0.0.1:6811/386250626] [v2:127.0.0.1:6812/386250626,v1:127.0.0.1:6813/386250626] exists,up 51efadca-5b36-49e3-b603-35075dc6e371 2024-04-25T07:40:05.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:40:05.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:40:05.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:40:05.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:40:05.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=139685 2024-04-25T07:40:05.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:40:05.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:40:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:40:05.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:40:05.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:05.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:05.697 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:05.697 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:05.697 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:40:05.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:40:05.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:40:05.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:40:05.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:40:05.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:40:05.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:40:05.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:40:05.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:40:05.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:40:05.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:40:05.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:40:05.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:40:05.699 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:40:05.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:40:05.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:40:05.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:40:05.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:40:05.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:40:05.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:40:05.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:40:05.729 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:05.728+0000 7f1b8ebde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:05.732 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:05.732+0000 7f1b8ebde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:05.733 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:05.732+0000 7f1b8ebde7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:06.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:40:06.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:40:06.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:40:06.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:40:06.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:40:06.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:06.171 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:40:06.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:40:06.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:06.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:06.522 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:06.520+0000 7f1b8ebde7c0 -1 Falling back to public interface 2024-04-25T07:40:06.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:07.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:07.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:07.629 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:40:07.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:40:07.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:07.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:08.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:09.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:09.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:09.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:40:09.049 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:40:09.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:09.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:09.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:10.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:10.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:10.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:40:10.452 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:40:10.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:10.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:10.507 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:10.508+0000 7f1b8ebde7c0 -1 osd.2 81 log_to_monitors true 2024-04-25T07:40:10.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:11.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:11.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:11.859 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:40:11.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:40:11.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:11.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:12.437 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 93 up_thru 93 down_at 82 last_clean_interval [68,81) [v2:127.0.0.1:6818/3796320512,v1:127.0.0.1:6819/3796320512] [v2:127.0.0.1:6820/3796320512,v1:127.0.0.1:6821/3796320512] exists,up 86c0edaf-9fed-4a03-a25d-5450ee8c066f 2024-04-25T07:40:12.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:40:12.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:40:12.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:40:12.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:40:12.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=140619 2024-04-25T07:40:12.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:40:12.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:12.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:40:12.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:40:12.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:12.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:12.443 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:12.443 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:12.443 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:40:12.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:40:12.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:40:12.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:40:12.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:40:12.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:40:12.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:40:12.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:40:12.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:40:12.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:40:12.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:40:12.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:40:12.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:40:12.445 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T07:40:12.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:40:12.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:40:12.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:40:12.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:40:12.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:40:12.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:40:12.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:40:12.472 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:12.472+0000 7f422eaac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:12.500 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:12.500+0000 7f422eaac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:12.501 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:12.500+0000 7f422eaac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:40:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:40:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:40:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:40:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:40:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:12.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:40:12.856 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:40:12.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:12.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:40:13.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:13.309 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:13.308+0000 7f422eaac7c0 -1 Falling back to public interface 2024-04-25T07:40:14.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:14.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:14.277 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:40:14.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:40:14.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:14.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:40:14.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:15.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:15.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:15.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:40:15.726 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:40:15.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:15.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:40:16.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:17.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:17.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:17.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:40:17.161 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:40:17.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:17.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:40:17.285 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:17.284+0000 7f422eaac7c0 -1 osd.3 83 log_to_monitors true 2024-04-25T07:40:17.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:18.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:18.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:18.724 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:40:18.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:40:18.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:18.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:40:19.375 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 98 up_thru 98 down_at 84 last_clean_interval [73,83) [v2:127.0.0.1:6826/65213390,v1:127.0.0.1:6827/65213390] [v2:127.0.0.1:6828/65213390,v1:127.0.0.1:6829/65213390] exists,up 8271f3ee-32c9-4e2f-a4cb-99671ce33ed0 2024-04-25T07:40:19.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:40:19.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:40:19.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:40:19.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:40:19.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=141583 2024-04-25T07:40:19.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:40:19.581 INFO:tasks.workunit.client.0.smithi102.stdout:hints = 1 2024-04-25T07:40:19.581 INFO:tasks.workunit.client.0.smithi102.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:40:19.581 INFO:tasks.workunit.client.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_142532 2024-04-25T07:40:19.581 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:40:19.582 INFO:tasks.workunit.client.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:40:20.581 INFO:tasks.workunit.client.0.smithi102.stdout: 1 16 112 96 0.374957 0.375 0.166924 0.154748 2024-04-25T07:40:21.581 INFO:tasks.workunit.client.0.smithi102.stdout: 2 16 218 202 0.39448 0.414062 0.115796 0.153369 2024-04-25T07:40:22.581 INFO:tasks.workunit.client.0.smithi102.stdout: 3 16 314 298 0.387966 0.375 0.176367 0.156003 2024-04-25T07:40:23.582 INFO:tasks.workunit.client.0.smithi102.stdout: 4 16 414 398 0.388617 0.390625 0.15386 0.155427 2024-04-25T07:40:24.582 INFO:tasks.workunit.client.0.smithi102.stdout: 5 16 504 488 0.381199 0.351562 0.221782 0.158489 2024-04-25T07:40:25.582 INFO:tasks.workunit.client.0.smithi102.stdout: 6 16 616 600 0.390575 0.4375 0.114039 0.15872 2024-04-25T07:40:26.582 INFO:tasks.workunit.client.0.smithi102.stdout: 7 16 723 707 0.394482 0.417969 0.181729 0.157236 2024-04-25T07:40:27.582 INFO:tasks.workunit.client.0.smithi102.stdout: 8 16 830 814 0.397412 0.417969 0.135214 0.156536 2024-04-25T07:40:28.583 INFO:tasks.workunit.client.0.smithi102.stdout: 9 16 928 912 0.395786 0.382812 0.192033 0.156241 2024-04-25T07:40:29.582 INFO:tasks.workunit.client.0.smithi102.stdout: 10 16 1043 1027 0.401123 0.449219 0.135505 0.154772 2024-04-25T07:40:30.582 INFO:tasks.workunit.client.0.smithi102.stdout: 11 16 1146 1130 0.401229 0.402344 0.165309 0.154261 2024-04-25T07:40:31.582 INFO:tasks.workunit.client.0.smithi102.stdout: 12 16 1258 1242 0.404247 0.4375 0.153059 0.154059 2024-04-25T07:40:32.583 INFO:tasks.workunit.client.0.smithi102.stdout: 13 16 1354 1338 0.401993 0.375 0.192789 0.154083 2024-04-25T07:40:33.583 INFO:tasks.workunit.client.0.smithi102.stdout: 14 16 1459 1443 0.402572 0.410156 0.167475 0.154699 2024-04-25T07:40:34.583 INFO:tasks.workunit.client.0.smithi102.stdout: 15 16 1555 1539 0.400731 0.375 0.221457 0.154811 2024-04-25T07:40:35.583 INFO:tasks.workunit.client.0.smithi102.stdout: 16 16 1647 1631 0.398142 0.359375 0.152608 0.156417 2024-04-25T07:40:36.583 INFO:tasks.workunit.client.0.smithi102.stdout: 17 16 1754 1738 0.399306 0.417969 0.192332 0.155838 2024-04-25T07:40:37.583 INFO:tasks.workunit.client.0.smithi102.stdout: 18 16 1871 1855 0.402509 0.457031 0.109648 0.154666 2024-04-25T07:40:38.584 INFO:tasks.workunit.client.0.smithi102.stdout: 19 16 1978 1962 0.40332 0.417969 0.210983 0.15398 2024-04-25T07:40:39.584 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:40:39.588351+0000 min lat: 0.0777321 max lat: 0.359089 avg lat: 0.154167 2024-04-25T07:40:39.584 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:40:39.584 INFO:tasks.workunit.client.0.smithi102.stdout: 20 16 2088 2072 0.404636 0.429688 0.118599 0.154167 2024-04-25T07:40:40.584 INFO:tasks.workunit.client.0.smithi102.stdout: 21 16 2161 2145 0.398944 0.285156 0.301466 0.155965 2024-04-25T07:40:41.584 INFO:tasks.workunit.client.0.smithi102.stdout: 22 16 2266 2250 0.399452 0.410156 0.174503 0.155818 2024-04-25T07:40:42.584 INFO:tasks.workunit.client.0.smithi102.stdout: 23 16 2367 2351 0.399236 0.394531 0.188388 0.156137 2024-04-25T07:40:43.584 INFO:tasks.workunit.client.0.smithi102.stdout: 24 16 2479 2463 0.400828 0.4375 0.131435 0.155509 2024-04-25T07:40:44.584 INFO:tasks.workunit.client.0.smithi102.stdout: 25 16 2578 2562 0.400262 0.386719 0.155307 0.155462 2024-04-25T07:40:45.584 INFO:tasks.workunit.client.0.smithi102.stdout: 26 16 2677 2661 0.399739 0.386719 0.148765 0.155518 2024-04-25T07:40:46.584 INFO:tasks.workunit.client.0.smithi102.stdout: 27 16 2749 2733 0.395348 0.28125 0.173462 0.157678 2024-04-25T07:40:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: 28 16 2853 2837 0.395736 0.40625 0.144018 0.157619 2024-04-25T07:40:48.585 INFO:tasks.workunit.client.0.smithi102.stdout: 29 16 2950 2934 0.395154 0.378906 0.151814 0.157427 2024-04-25T07:40:49.585 INFO:tasks.workunit.client.0.smithi102.stdout: 30 16 3062 3046 0.396563 0.4375 0.140616 0.157098 2024-04-25T07:40:50.585 INFO:tasks.workunit.client.0.smithi102.stdout:Total time run: 30.0858 2024-04-25T07:40:50.585 INFO:tasks.workunit.client.0.smithi102.stdout:Total writes made: 3062 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Write size: 4096 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Object size: 4096 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Bandwidth (MB/sec): 0.397561 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Bandwidth: 0.0409271 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Max bandwidth (MB/sec): 0.457031 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Min bandwidth (MB/sec): 0.28125 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Average IOPS: 101 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev IOPS: 10.4773 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Max IOPS: 117 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Min IOPS: 72 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Average Latency(s): 0.157029 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Latency(s): 0.0450628 2024-04-25T07:40:50.586 INFO:tasks.workunit.client.0.smithi102.stdout:Max latency(s): 0.495293 2024-04-25T07:40:50.587 INFO:tasks.workunit.client.0.smithi102.stdout:Min latency(s): 0.0777321 2024-04-25T07:40:50.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-04-25T07:40:50.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:40:50.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:40:50.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:40:50.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:40:50.648 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:40:50.648 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:40:50.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:40:50.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:40:50.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:40:50.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:40:50.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:40:50.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:40:50.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:40:50.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:40:50.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:40:51.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:40:51.192 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:40:51.192 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T07:40:51.192 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T07:40:51.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:40:51.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:40:51.192 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:40:51.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=369367187469 2024-04-25T07:40:51.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 369367187469 2024-04-25T07:40:51.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-369367187469' 2024-04-25T07:40:51.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:40:51.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:40:51.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=382252089356 2024-04-25T07:40:51.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 382252089356 2024-04-25T07:40:51.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-369367187469 1-382252089356' 2024-04-25T07:40:51.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:40:51.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:40:51.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=399431958539 2024-04-25T07:40:51.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 399431958539 2024-04-25T07:40:51.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-399431958539' 2024-04-25T07:40:51.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:40:51.643 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:40:51.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=420906795017 2024-04-25T07:40:51.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 420906795017 2024-04-25T07:40:51.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-369367187469 1-382252089356 2-399431958539 3-420906795017' 2024-04-25T07:40:51.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:40:51.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-369367187469 2024-04-25T07:40:51.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:40:51.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:40:51.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-369367187469 2024-04-25T07:40:51.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:40:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=369367187469 2024-04-25T07:40:51.792 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 369367187469 2024-04-25T07:40:51.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 369367187469' 2024-04-25T07:40:51.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:40:52.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 369367187469 -lt 369367187469 2024-04-25T07:40:52.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:40:52.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-382252089356 2024-04-25T07:40:52.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:40:52.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:40:52.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-382252089356 2024-04-25T07:40:52.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:40:52.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=382252089356 2024-04-25T07:40:52.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 382252089356' 2024-04-25T07:40:52.232 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 382252089356 2024-04-25T07:40:52.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:40:52.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 382252089356 -lt 382252089356 2024-04-25T07:40:52.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:40:52.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-399431958539 2024-04-25T07:40:52.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:40:52.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:40:52.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-399431958539 2024-04-25T07:40:52.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:40:52.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=399431958539 2024-04-25T07:40:52.684 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 399431958539 2024-04-25T07:40:52.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 399431958539' 2024-04-25T07:40:52.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:40:53.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 399431958539 -lt 399431958539 2024-04-25T07:40:53.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:40:53.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-420906795017 2024-04-25T07:40:53.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:40:53.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:40:53.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-420906795017 2024-04-25T07:40:53.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:40:53.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=420906795017 2024-04-25T07:40:53.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 420906795017' 2024-04-25T07:40:53.115 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 420906795017 2024-04-25T07:40:53.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:40:53.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 420906795017 -lt 420906795017 2024-04-25T07:40:53.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:40:53.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:40:53.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:40:54.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:40:54.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:40:54.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:40:54.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:40:54.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:40:54.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:40:54.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:40:54.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:40:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:40:54.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:40:54.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:40:54.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:40:55.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:40:55.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:40:55.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:40:55.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:40:55.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:55.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:40:55.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:40:55.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:40:55.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:40:55.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:40:55.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:40:55.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:40:55.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:40:55.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:40:55.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:40:55.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:40:55.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:40:55.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:40:55.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:40:55.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:40:55.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:40:55.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:40:55.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:40:55.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:40:55.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:40:55.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:40:55.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:40:55.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:55.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:40:55.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95853 2024-04-25T07:40:55.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:40:55.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:40:55.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:40:55.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:40:55.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:55.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:40:55.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:55.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:40:55.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:40:55.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:40:55.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:40:55.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:40:55.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:40:55.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:40:55.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:40:55.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:40:55.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:40:55.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:40:55.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:40:55.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:40:55.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:40:55.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:40:55.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:40:55.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:40:55.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:40:55.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:40:55.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:40:55.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:40:55.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:40:55.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:55.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:40:55.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95853 2024-04-25T07:40:55.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:40:55.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:40:55.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:40:55.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:40:55.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:55.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:40:55.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.95853 2024-04-25T07:40:55.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:40:55.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:40:55.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:40:55.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:40:55.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-04-25T07:40:55.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:55.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-04-25T07:40:55.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-04-25T07:40:55.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-04-25T07:40:55.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:40:55.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-04-25T07:40:55.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-04-25T07:40:55.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=865fbc68-a446-43ce-a4d0-92435354e202 --auth-supported=none ' 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:40:55.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:55.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:40:55.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:40:55.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:40:55.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:40:55.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:56.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:40:56.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:56.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:56.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:56.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:56.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:40:56.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:40:56.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:40:56.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:40:56.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:40:56.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:40:56.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:40:56.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:40:56.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:40:56.633 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:40:56.633 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:40:56.633 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:40:56.633 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:40:56.633 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:56.633 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:40:56.634 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95853/ceph-mon.a.asok 2024-04-25T07:40:56.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:40:56.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95853/ceph-mon.a.asok config get fsid 2024-04-25T07:40:56.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:40:56.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:40:56.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:40:56.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:40:56.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:40:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:40:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:40:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:40:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:40:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:40:56.761 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95853/ceph-mon.a.asok 2024-04-25T07:40:56.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:40:56.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95853/ceph-mon.a.asok config get mon_host 2024-04-25T07:40:56.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:40:56.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:56.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:40:56.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:40:56.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:40:56.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:40:56.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:40:57.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:40:57.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:57.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:57.804 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:57.804 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:57.804 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:40:57.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:40:57.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:40:57.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:40:57.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:40:57.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:57.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:40:57.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=865fbc68-a446-43ce-a4d0-92435354e202 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:57.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:40:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:40:57.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:40:57.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:40:57.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:40:57.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:40:57.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:40:57.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a376a77f-aa28-4d13-9279-6a76e4835258 2024-04-25T07:40:57.847 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 a376a77f-aa28-4d13-9279-6a76e4835258 2024-04-25T07:40:57.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a376a77f-aa28-4d13-9279-6a76e4835258' 2024-04-25T07:40:57.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:40:57.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAJCSpmJd/VMxAAwH7qrlfLta4OsU9TtlfVIw== 2024-04-25T07:40:57.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAJCSpmJd/VMxAAwH7qrlfLta4OsU9TtlfVIw=="}' 2024-04-25T07:40:57.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a376a77f-aa28-4d13-9279-6a76e4835258 -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:40:58.209 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:40:58.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:40:58.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=865fbc68-a446-43ce-a4d0-92435354e202 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAJCSpmJd/VMxAAwH7qrlfLta4OsU9TtlfVIw== --osd-uuid a376a77f-aa28-4d13-9279-6a76e4835258 2024-04-25T07:40:58.261 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:58.263+0000 7f2b3ef0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:58.286 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:58.287+0000 7f2b3ef0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:58.287 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:58.287+0000 7f2b3ef0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:58.287 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:58.291+0000 7f2b3ef0e7c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:40:58.287 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:58.291+0000 7f2b3ef0e7c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:40:58.287 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:58.291+0000 7f2b3ef0e7c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:40:58.326 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:40:58.327+0000 7f2b3ef0e7c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:41:06.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:41:06.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:41:06.746 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:41:06.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:41:06.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:41:07.310 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:41:07.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:41:07.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:41:07.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=865fbc68-a446-43ce-a4d0-92435354e202 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:41:07.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:41:07.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:41:07.347 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:41:07.347+0000 7fd559ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:07.349 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:41:07.351+0000 7fd559ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:07.350 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:41:07.351+0000 7fd559ec57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:07.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:41:07.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:41:07.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:41:07.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:41:07.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:41:07.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:07.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:41:07.753 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:41:07.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:07.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:08.158 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:41:08.159+0000 7fd559ec57c0 -1 Falling back to public interface 2024-04-25T07:41:08.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:09.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:09.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:09.193 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:41:09.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:41:09.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:09.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:09.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:10.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:10.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:10.605 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:41:10.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:41:10.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:10.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:11.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:12.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:12.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:12.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:41:12.022 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:41:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:12.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:12.317 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:41:12.319+0000 7fd559ec57c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:41:12.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:13.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:13.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:13.443 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:41:13.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:41:13.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:13.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:14.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:15.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:15.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:15.458 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:41:15.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:41:15.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:15.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:16.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:17.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:17.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:17.259 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:41:17.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:41:17.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:17.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:17.693 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/159490339,v1:127.0.0.1:6803/159490339] [v2:127.0.0.1:6804/159490339,v1:127.0.0.1:6805/159490339] exists,up a376a77f-aa28-4d13-9279-6a76e4835258 2024-04-25T07:41:17.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:41:17.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:41:17.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:41:17.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:41:17.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=144403 2024-04-25T07:41:17.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-04-25T07:41:22.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-04-25T07:41:22.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:41:23.340 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T07:41:23.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:41:24.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-04-25T07:41:24.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-04-25T07:41:24.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-04-25T07:41:24.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-04-25T07:41:24.462 INFO:tasks.workunit.client.0.smithi102.stdout:hints = 1 2024-04-25T07:41:24.465 INFO:tasks.workunit.client.0.smithi102.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-04-25T07:41:24.465 INFO:tasks.workunit.client.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_145686 2024-04-25T07:41:24.511 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:41:24.511 INFO:tasks.workunit.client.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:41:25.511 INFO:tasks.workunit.client.0.smithi102.stdout: 1 16 17 1 3.99923 4 0.312517 0.312517 2024-04-25T07:41:26.511 INFO:tasks.workunit.client.0.smithi102.stdout: 2 16 33 17 33.9948 64 1.09467 1.18942 2024-04-25T07:41:27.511 INFO:tasks.workunit.client.0.smithi102.stdout: 3 16 49 33 43.9933 64 1.06043 1.13964 2024-04-25T07:41:28.511 INFO:tasks.workunit.client.0.smithi102.stdout: 4 16 65 49 48.9925 64 0.926938 1.08102 2024-04-25T07:41:29.511 INFO:tasks.workunit.client.0.smithi102.stdout: 5 16 81 65 51.9925 64 1.32052 1.10278 2024-04-25T07:41:30.512 INFO:tasks.workunit.client.0.smithi102.stdout: 6 16 96 80 53.3259 60 1.49167 1.17649 2024-04-25T07:41:31.511 INFO:tasks.workunit.client.0.smithi102.stdout: 7 16 112 96 54.8494 64 1.02069 1.14976 2024-04-25T07:41:32.512 INFO:tasks.workunit.client.0.smithi102.stdout: 8 16 128 112 55.9921 64 0.955699 1.1201 2024-04-25T07:41:33.512 INFO:tasks.workunit.client.0.smithi102.stdout: 9 16 144 128 56.8809 64 1.05714 1.1129 2024-04-25T07:41:34.512 INFO:tasks.workunit.client.0.smithi102.stdout: 10 16 145 129 51.5926 4 1.05231 1.11243 2024-04-25T07:41:35.512 INFO:tasks.workunit.client.0.smithi102.stdout:Total time run: 10.2112 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Total writes made: 145 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Write size: 4194304 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Object size: 4194304 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Bandwidth (MB/sec): 56.8006 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Bandwidth: 25.1184 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Max bandwidth (MB/sec): 64 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Min bandwidth (MB/sec): 4 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Average IOPS: 14 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev IOPS: 6.2796 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Max IOPS: 16 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Min IOPS: 1 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Average Latency(s): 1.11035 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Latency(s): 0.17513 2024-04-25T07:41:35.513 INFO:tasks.workunit.client.0.smithi102.stdout:Max latency(s): 1.50493 2024-04-25T07:41:35.514 INFO:tasks.workunit.client.0.smithi102.stdout:Min latency(s): 0.312517 2024-04-25T07:41:35.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-04-25T07:41:45.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-04-25T07:41:45.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-04-25T07:41:45.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1003487232 2024-04-25T07:41:45.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-04-25T07:41:45.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-04-25T07:41:45.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=0 2024-04-25T07:41:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-04-25T07:41:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 0 -eq 0 2024-04-25T07:41:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ 1 -le 5 ]] 2024-04-25T07:41:45.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-04-25T07:41:45.968 INFO:tasks.workunit.client.0.smithi102.stdout:hints = 1 2024-04-25T07:41:45.970 INFO:tasks.workunit.client.0.smithi102.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-04-25T07:41:45.970 INFO:tasks.workunit.client.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_145759 2024-04-25T07:41:46.001 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:41:46.001 INFO:tasks.workunit.client.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:41:47.001 INFO:tasks.workunit.client.0.smithi102.stdout: 1 16 17 1 3.99925 4 0.116432 0.116432 2024-04-25T07:41:48.001 INFO:tasks.workunit.client.0.smithi102.stdout: 2 16 33 17 33.9936 64 1.34477 1.31702 2024-04-25T07:41:49.002 INFO:tasks.workunit.client.0.smithi102.stdout: 3 16 64 48 63.9895 124 0.884857 0.98304 2024-04-25T07:41:50.002 INFO:tasks.workunit.client.0.smithi102.stdout: 4 16 80 64 63.9895 64 0.952433 0.975363 2024-04-25T07:41:51.002 INFO:tasks.workunit.client.0.smithi102.stdout: 5 16 81 65 51.9917 4 0.85122 0.973453 2024-04-25T07:41:52.002 INFO:tasks.workunit.client.0.smithi102.stdout: 6 16 97 81 53.9914 64 1.3071 1.03389 2024-04-25T07:41:53.002 INFO:tasks.workunit.client.0.smithi102.stdout: 7 16 113 97 55.4198 64 0.990042 1.03198 2024-04-25T07:41:54.002 INFO:tasks.workunit.client.0.smithi102.stdout: 8 16 129 113 56.4908 64 1.65481 1.06612 2024-04-25T07:41:55.003 INFO:tasks.workunit.client.0.smithi102.stdout: 9 16 129 113 50.2141 0 - 1.06612 2024-04-25T07:41:56.003 INFO:tasks.workunit.client.0.smithi102.stdout: 10 16 129 113 45.1926 0 - 1.06612 2024-04-25T07:41:57.003 INFO:tasks.workunit.client.0.smithi102.stdout: 11 16 129 113 41.0843 0 - 1.06612 2024-04-25T07:41:58.003 INFO:tasks.workunit.client.0.smithi102.stdout:Total time run: 11.9756 2024-04-25T07:41:58.003 INFO:tasks.workunit.client.0.smithi102.stdout:Total writes made: 129 2024-04-25T07:41:58.003 INFO:tasks.workunit.client.0.smithi102.stdout:Write size: 4194304 2024-04-25T07:41:58.003 INFO:tasks.workunit.client.0.smithi102.stdout:Object size: 4194304 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Bandwidth (MB/sec): 43.0876 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Bandwidth: 41.6112 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Max bandwidth (MB/sec): 124 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Min bandwidth (MB/sec): 0 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Average IOPS: 10 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev IOPS: 10.4028 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Max IOPS: 31 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Min IOPS: 0 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Average Latency(s): 1.45331 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Latency(s): 1.05875 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Max latency(s): 4.20118 2024-04-25T07:41:58.004 INFO:tasks.workunit.client.0.smithi102.stdout:Min latency(s): 0.116432 2024-04-25T07:41:58.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-04-25T07:42:08.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-04-25T07:42:08.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-04-25T07:42:08.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=127926272 2024-04-25T07:42:08.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-04-25T07:42:08.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-04-25T07:42:08.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=351141888 2024-04-25T07:42:08.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-04-25T07:42:08.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 351141888 -eq 0 2024-04-25T07:42:08.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-04-25T07:42:08.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 351141888 -gt 0 2024-04-25T07:42:08.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 144403 2024-04-25T07:42:08.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-04-25T07:42:09.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 144403 2024-04-25T07:42:09.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (144403) - No such process 2024-04-25T07:42:09.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-04-25T07:42:09.835 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T07:42:09.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-04-25T07:42:09.878 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:42:12.660 INFO:tasks.workunit.client.0.smithi102.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-04-25T07:42:12.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-04-25T07:42:12.714 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:42:13.745 INFO:tasks.workunit.client.0.smithi102.stdout:1 : device size 0xffffe000 : using 0x1caf7000(459 MiB) 2024-04-25T07:42:13.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-04-25T07:42:15.564 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:42:15.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:42:15.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:42:15.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=865fbc68-a446-43ce-a4d0-92435354e202 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:42:15.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:42:15.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:42:15.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:42:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:42:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:42:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:42:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:42:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:42:15.634 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:42:15.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:42:15.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=865fbc68-a446-43ce-a4d0-92435354e202 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:42:15.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:42:15.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:42:15.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:42:15.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:42:15.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:42:15.677 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:15.677+0000 7f3a343c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:15.679 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:15.677+0000 7f3a343c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:15.680 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:15.681+0000 7f3a343c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:16.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:42:16.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:42:16.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:42:16.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:42:16.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:42:16.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:16.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:42:16.038 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:42:16.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:16.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:16.225 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:16.225+0000 7f3a343c77c0 -1 Falling back to public interface 2024-04-25T07:42:16.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:17.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:17.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:17.463 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:42:17.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:42:17.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:17.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:17.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:18.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:18.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:18.896 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:42:18.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:42:18.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:18.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:19.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:20.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:20.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:20.323 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:42:20.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:42:20.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:20.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:20.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:21.174 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:21.173+0000 7f3a343c77c0 -1 osd.0 10 log_to_monitors true 2024-04-25T07:42:21.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:21.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:21.757 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:42:21.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:42:21.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:21.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:22.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:23.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:23.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:23.455 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:42:23.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:42:23.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:23.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:23.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:24.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:24.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:24.921 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:42:24.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:42:24.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:24.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:25.446 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 13 up_thru 7 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/3218273018,v1:127.0.0.1:6803/3218273018] [v2:127.0.0.1:6804/3218273018,v1:127.0.0.1:6805/3218273018] exists,up a376a77f-aa28-4d13-9279-6a76e4835258 2024-04-25T07:42:25.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:42:25.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:42:25.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:42:25.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:42:25.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=146984 2024-04-25T07:42:25.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-04-25T07:42:25.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:42:25.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:42:25.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:42:25.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:42:25.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:42:25.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:42:25.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:42:25.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:42:25.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:42:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:42:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:42:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:42:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:42:25.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:42:25.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:42:25.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T07:42:25.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:42:25.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:42:25.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:42:26.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574851 2024-04-25T07:42:26.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574851 2024-04-25T07:42:26.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-55834574851' 2024-04-25T07:42:26.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:42:26.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-55834574851 2024-04-25T07:42:26.159 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:42:26.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:42:26.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-55834574851 2024-04-25T07:42:26.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:42:26.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574851 2024-04-25T07:42:26.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 55834574851' 2024-04-25T07:42:26.161 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 55834574851 2024-04-25T07:42:26.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:42:26.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574851 -lt 55834574851 2024-04-25T07:42:26.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:42:26.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:42:26.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:42:27.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:42:27.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:42:27.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:42:27.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:42:27.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:42:27.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:42:27.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:42:27.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:42:27.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:42:27.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:42:27.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:42:27.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:42:28.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:42:28.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:42:28.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:42:28.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:42:28.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:42:28.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:42:28.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:42:28.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:42:28.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:42:28.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:42:28.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:42:28.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:42:28.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:42:28.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:42:28.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:42:28.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:42:28.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:42:28.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:42:28.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:42:28.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:28.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:42:28.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:42:28.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:28.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:42:28.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:42:28.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:42:28.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:42:28.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:28.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:42:28.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95853 2024-04-25T07:42:28.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:42:28.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:42:28.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:42:28.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:42:28.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:42:28.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:42:28.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:42:28.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:42:28.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:42:28.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:42:28.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:42:28.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:42:28.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:42:28.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:42:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:42:28.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:42:28.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:42:28.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:42:28.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:42:28.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:42:28.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:42:28.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:28.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:42:28.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:42:28.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:42:28.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:42:28.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:42:28.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:42:28.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:42:28.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:28.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:42:28.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95853 2024-04-25T07:42:28.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:42:28.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:42:28.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:42:28.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:42:28.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:28.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:42:28.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.95853 2024-04-25T07:42:28.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:42:28.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:42:28.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:42:28.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:42:28.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-04-25T07:42:28.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:42:28.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-04-25T07:42:28.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-04-25T07:42:28.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-04-25T07:42:28.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:42:28.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-04-25T07:42:28.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-04-25T07:42:28.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-04-25T07:42:28.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=f23270db-79ee-4eca-8e4f-3cb4e99d3b21 --auth-supported=none ' 2024-04-25T07:42:28.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:42:28.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-04-25T07:42:28.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:42:28.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:42:28.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:42:28.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:42:28.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:28.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:28.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:28.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:28.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:42:28.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:42:28.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:42:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:42:29.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:42:29.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:42:29.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:42:29.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:42:29.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:42:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:42:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:42:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:42:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:42:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:42:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95853/ceph-mon.a.asok 2024-04-25T07:42:29.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:42:29.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95853/ceph-mon.a.asok config get fsid 2024-04-25T07:42:29.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:42:29.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:42:29.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:42:29.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:42:29.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:42:29.140 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:42:29.140 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:42:29.140 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:42:29.141 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:42:29.141 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:29.141 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:42:29.141 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95853/ceph-mon.a.asok 2024-04-25T07:42:29.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:42:29.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95853/ceph-mon.a.asok config get mon_host 2024-04-25T07:42:29.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:42:29.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:42:29.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:42:29.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:42:29.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:42:29.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:42:29.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:42:30.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:42:30.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:30.192 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:30.192 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:30.192 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:30.192 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:42:30.192 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:42:30.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:42:30.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:42:30.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:42:30.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:42:30.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:42:30.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:42:30.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:42:30.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:42:30.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=865fbc68-a446-43ce-a4d0-92435354e202 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=f23270db-79ee-4eca-8e4f-3cb4e99d3b21 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:42:30.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:42:30.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:42:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:42:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:42:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:42:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:42:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:42:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:42:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:42:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:42:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:42:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:42:30.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:42:30.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:42:30.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb9c7d09-7497-47e7-9feb-d104b6518f3f 2024-04-25T07:42:30.278 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 eb9c7d09-7497-47e7-9feb-d104b6518f3f 2024-04-25T07:42:30.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 eb9c7d09-7497-47e7-9feb-d104b6518f3f' 2024-04-25T07:42:30.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:42:30.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBmCSpmcbJOEhAAFoeERfIgtLeNuYiQgdivCw== 2024-04-25T07:42:30.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBmCSpmcbJOEhAAFoeERfIgtLeNuYiQgdivCw=="}' 2024-04-25T07:42:30.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb9c7d09-7497-47e7-9feb-d104b6518f3f -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:42:30.637 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:42:30.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:42:30.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=865fbc68-a446-43ce-a4d0-92435354e202 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=f23270db-79ee-4eca-8e4f-3cb4e99d3b21 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBmCSpmcbJOEhAAFoeERfIgtLeNuYiQgdivCw== --osd-uuid eb9c7d09-7497-47e7-9feb-d104b6518f3f 2024-04-25T07:42:30.690 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:30.689+0000 7feb843197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:30.706 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:30.705+0000 7feb843197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:30.707 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:30.705+0000 7feb843197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:30.707 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:30.709+0000 7feb843197c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:42:30.708 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:30.709+0000 7feb843197c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:42:30.708 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:30.709+0000 7feb843197c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:42:30.746 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:30.745+0000 7feb843197c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:42:38.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:42:38.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:42:38.938 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:42:38.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:42:38.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:42:39.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:42:39.498 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:42:39.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:42:39.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=865fbc68-a446-43ce-a4d0-92435354e202 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=f23270db-79ee-4eca-8e4f-3cb4e99d3b21 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:42:39.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:42:39.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:42:39.540 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:39.541+0000 7f258e4057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:39.571 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:39.569+0000 7f258e4057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:39.572 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:39.573+0000 7f258e4057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:39.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:42:39.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:42:39.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:42:39.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:42:39.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:42:39.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:39.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:42:39.928 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:42:39.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:39.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:40.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:40.401 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:40.401+0000 7f258e4057c0 -1 Falling back to public interface 2024-04-25T07:42:41.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:41.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:41.358 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:42:41.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:42:41.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:41.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:41.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:42.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:42.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:42.800 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:42:42.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:42:42.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:42.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:43.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:44.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:44.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:44.237 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:42:44.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:42:44.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:44.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:44.459 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:42:44.456+0000 7f258e4057c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:42:44.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:45.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:45.688 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:42:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:42:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:46.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:47.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:47.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:47.409 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:42:47.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:42:47.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:47.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:48.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:49.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:49.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:49.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:42:49.355 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:42:49.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:49.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:49.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:50.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:50.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:50.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:42:50.831 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:42:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:51.240 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2206462149,v1:127.0.0.1:6803/2206462149] [v2:127.0.0.1:6804/2206462149,v1:127.0.0.1:6805/2206462149] exists,up eb9c7d09-7497-47e7-9feb-d104b6518f3f 2024-04-25T07:42:51.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:42:51.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:42:51.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:42:51.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:42:51.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=149652 2024-04-25T07:42:51.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-04-25T07:42:56.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-04-25T07:42:56.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:42:56.780 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T07:42:56.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:42:57.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:42:57.880 INFO:tasks.workunit.client.0.smithi102.stdout:hints = 1 2024-04-25T07:42:57.880 INFO:tasks.workunit.client.0.smithi102.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:42:57.880 INFO:tasks.workunit.client.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_150703 2024-04-25T07:42:57.880 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:42:57.880 INFO:tasks.workunit.client.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:42:58.880 INFO:tasks.workunit.client.0.smithi102.stdout: 1 16 96 80 0.312481 0.3125 0.134073 0.181508 2024-04-25T07:42:59.880 INFO:tasks.workunit.client.0.smithi102.stdout: 2 16 192 176 0.343713 0.375 0.101536 0.178412 2024-04-25T07:43:00.880 INFO:tasks.workunit.client.0.smithi102.stdout: 3 16 306 290 0.377564 0.445312 0.208943 0.159753 2024-04-25T07:43:01.880 INFO:tasks.workunit.client.0.smithi102.stdout: 4 16 416 400 0.390579 0.429688 0.118398 0.158164 2024-04-25T07:43:02.881 INFO:tasks.workunit.client.0.smithi102.stdout: 5 16 530 514 0.401511 0.445312 0.142261 0.150694 2024-04-25T07:43:03.881 INFO:tasks.workunit.client.0.smithi102.stdout: 6 16 656 640 0.416615 0.492188 0.141553 0.14948 2024-04-25T07:43:04.881 INFO:tasks.workunit.client.0.smithi102.stdout: 7 16 754 738 0.411776 0.382812 0.116999 0.148972 2024-04-25T07:43:05.881 INFO:tasks.workunit.client.0.smithi102.stdout: 8 16 864 848 0.414009 0.429688 0.151735 0.150345 2024-04-25T07:43:06.881 INFO:tasks.workunit.client.0.smithi102.stdout: 9 16 978 962 0.41748 0.445312 0.108752 0.147663 2024-04-25T07:43:07.881 INFO:tasks.workunit.client.0.smithi102.stdout: 10 16 1104 1088 0.424946 0.492188 0.10014 0.146854 2024-04-25T07:43:08.881 INFO:tasks.workunit.client.0.smithi102.stdout: 11 16 1216 1200 0.426082 0.4375 0.159585 0.145283 2024-04-25T07:43:09.881 INFO:tasks.workunit.client.0.smithi102.stdout: 12 16 1314 1298 0.422473 0.382812 0.158935 0.146728 2024-04-25T07:43:10.882 INFO:tasks.workunit.client.0.smithi102.stdout: 13 16 1440 1424 0.42783 0.492188 0.125595 0.145475 2024-04-25T07:43:11.882 INFO:tasks.workunit.client.0.smithi102.stdout: 14 16 1538 1522 0.42461 0.382812 0.117122 0.146025 2024-04-25T07:43:12.882 INFO:tasks.workunit.client.0.smithi102.stdout: 15 16 1666 1650 0.429631 0.5 0.150583 0.144311 2024-04-25T07:43:13.882 INFO:tasks.workunit.client.0.smithi102.stdout: 16 16 1776 1760 0.429632 0.429688 0.151474 0.144936 2024-04-25T07:43:14.882 INFO:tasks.workunit.client.0.smithi102.stdout: 17 16 1890 1874 0.43055 0.445312 0.133814 0.144137 2024-04-25T07:43:15.882 INFO:tasks.workunit.client.0.smithi102.stdout: 18 16 2002 1986 0.430932 0.4375 0.125387 0.144151 2024-04-25T07:43:16.883 INFO:tasks.workunit.client.0.smithi102.stdout: 19 16 2128 2112 0.434152 0.492188 0.109404 0.14341 2024-04-25T07:43:17.883 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:43:17.887138+0000 min lat: 0.0920087 max lat: 0.360004 avg lat: 0.143415 2024-04-25T07:43:17.883 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:43:17.883 INFO:tasks.workunit.client.0.smithi102.stdout: 20 16 2240 2224 0.434316 0.4375 0.142823 0.143415 2024-04-25T07:43:18.883 INFO:tasks.workunit.client.0.smithi102.stdout: 21 16 2354 2338 0.434835 0.445312 0.167401 0.143048 2024-04-25T07:43:19.883 INFO:tasks.workunit.client.0.smithi102.stdout: 22 16 2466 2450 0.434954 0.4375 0.117138 0.143347 2024-04-25T07:43:20.883 INFO:tasks.workunit.client.0.smithi102.stdout: 23 16 2592 2576 0.43744 0.492188 0.150296 0.142761 2024-04-25T07:43:21.883 INFO:tasks.workunit.client.0.smithi102.stdout: 24 16 2690 2674 0.435162 0.382812 0.142268 0.143118 2024-04-25T07:43:22.883 INFO:tasks.workunit.client.0.smithi102.stdout: 25 16 2802 2786 0.435253 0.4375 0.133868 0.142748 2024-04-25T07:43:23.884 INFO:tasks.workunit.client.0.smithi102.stdout: 26 16 2914 2898 0.435337 0.4375 0.142189 0.143057 2024-04-25T07:43:24.884 INFO:tasks.workunit.client.0.smithi102.stdout: 27 16 3040 3024 0.43744 0.492188 0.108832 0.142448 2024-04-25T07:43:25.884 INFO:tasks.workunit.client.0.smithi102.stdout: 28 16 3152 3136 0.437441 0.4375 0.125387 0.142772 2024-04-25T07:43:26.884 INFO:tasks.workunit.client.0.smithi102.stdout: 29 16 3264 3248 0.437441 0.4375 0.12488 0.14251 2024-04-25T07:43:27.884 INFO:tasks.workunit.client.0.smithi102.stdout: 30 16 3376 3360 0.437441 0.4375 0.15171 0.142741 2024-04-25T07:43:28.884 INFO:tasks.workunit.client.0.smithi102.stdout:Total time run: 30.1202 2024-04-25T07:43:28.885 INFO:tasks.workunit.client.0.smithi102.stdout:Total writes made: 3376 2024-04-25T07:43:28.885 INFO:tasks.workunit.client.0.smithi102.stdout:Write size: 4096 2024-04-25T07:43:28.885 INFO:tasks.workunit.client.0.smithi102.stdout:Object size: 4096 2024-04-25T07:43:28.885 INFO:tasks.workunit.client.0.smithi102.stdout:Bandwidth (MB/sec): 0.437829 2024-04-25T07:43:28.885 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Bandwidth: 0.0430849 2024-04-25T07:43:28.885 INFO:tasks.workunit.client.0.smithi102.stdout:Max bandwidth (MB/sec): 0.5 2024-04-25T07:43:28.885 INFO:tasks.workunit.client.0.smithi102.stdout:Min bandwidth (MB/sec): 0.3125 2024-04-25T07:43:28.885 INFO:tasks.workunit.client.0.smithi102.stdout:Average IOPS: 112 2024-04-25T07:43:28.886 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev IOPS: 11.0297 2024-04-25T07:43:28.886 INFO:tasks.workunit.client.0.smithi102.stdout:Max IOPS: 128 2024-04-25T07:43:28.886 INFO:tasks.workunit.client.0.smithi102.stdout:Min IOPS: 80 2024-04-25T07:43:28.886 INFO:tasks.workunit.client.0.smithi102.stdout:Average Latency(s): 0.142703 2024-04-25T07:43:28.886 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Latency(s): 0.0336243 2024-04-25T07:43:28.886 INFO:tasks.workunit.client.0.smithi102.stdout:Max latency(s): 0.360004 2024-04-25T07:43:28.886 INFO:tasks.workunit.client.0.smithi102.stdout:Min latency(s): 0.0920087 2024-04-25T07:43:28.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-04-25T07:43:33.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-04-25T07:43:33.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-04-25T07:43:34.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-04-25T07:43:34.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-04-25T07:43:34.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-04-25T07:43:34.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-04-25T07:43:34.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-04-25T07:43:34.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-04-25T07:43:34.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-04-25T07:43:34.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4281036800 2024-04-25T07:43:34.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 149652 2024-04-25T07:43:34.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-04-25T07:43:35.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 149652 2024-04-25T07:43:35.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (149652) - No such process 2024-04-25T07:43:35.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-04-25T07:43:35.816 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T07:43:35.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-04-25T07:43:35.868 INFO:tasks.workunit.client.0.smithi102.stdout:allocmap bluestore.allocmap 2024-04-25T07:43:38.652 INFO:tasks.workunit.client.0.smithi102.stdout:allocmap success 2024-04-25T07:43:38.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:446: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:43:41.228 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:43:41.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:448: TEST_bluestore_expand: requested_space=4294967296 2024-04-25T07:43:41.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:449: TEST_bluestore_expand: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-04-25T07:43:41.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:450: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-04-25T07:43:41.294 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:43:43.046 INFO:tasks.workunit.client.0.smithi102.stdout:1 : device size 0x3fffe000 : using 0x2300000(35 MiB) 2024-04-25T07:43:43.046 INFO:tasks.workunit.client.0.smithi102.stdout:2 : device size 0x100000000 : using 0xd49000(13 MiB) 2024-04-25T07:43:43.046 INFO:tasks.workunit.client.0.smithi102.stdout:Expanding DB/WAL... 2024-04-25T07:43:43.046 INFO:tasks.workunit.client.0.smithi102.stdout:1 : expanding to 0x1073741824 2024-04-25T07:43:43.073 INFO:tasks.workunit.client.0.smithi102.stdout:1 : size label updated to 1073741824 2024-04-25T07:43:43.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:453: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:43:46.152 INFO:tasks.workunit.client.0.smithi102.stdout:fsck success 2024-04-25T07:43:46.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:456: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-04-25T07:43:46.209 INFO:tasks.workunit.client.0.smithi102.stdout:qfsck bluestore.quick-fsck 2024-04-25T07:43:48.492 INFO:tasks.workunit.client.0.smithi102.stdout:qfsck success 2024-04-25T07:43:48.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:458: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-04-25T07:43:48.536 INFO:tasks.workunit.client.0.smithi102.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:43:50.829 INFO:tasks.workunit.client.0.smithi102.stdout:1 : device size 0x3fffe000 : using 0x2300000(35 MiB) 2024-04-25T07:43:50.830 INFO:tasks.workunit.client.0.smithi102.stdout:2 : device size 0x100000000 : using 0xd49000(13 MiB) 2024-04-25T07:43:50.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:460: TEST_bluestore_expand: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:43:50.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:43:50.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:43:50.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:43:50.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:43:50.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:43:50.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=865fbc68-a446-43ce-a4d0-92435354e202 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=f23270db-79ee-4eca-8e4f-3cb4e99d3b21 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:43:50.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:43:50.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:43:50.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' 2024-04-25T07:43:50.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:43:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:43:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:43:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:43:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:43:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:43:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:43:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:43:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:43:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:43:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:43:50.843 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:43:50.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=44195330-70f3-481f-b796-5e5724a3fb73 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=865fbc68-a446-43ce-a4d0-92435354e202 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=f23270db-79ee-4eca-8e4f-3cb4e99d3b21 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95853/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:43:50.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:43:50.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:43:50.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:43:50.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:43:50.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:43:50.886 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:43:50.887+0000 7fd77a17f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:50.889 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:43:50.891+0000 7fd77a17f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:50.891 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:43:50.891+0000 7fd77a17f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:51.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:43:51.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:43:51.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:43:51.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:43:51.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:43:51.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:51.271 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:43:51.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:43:51.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:51.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:51.666 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:43:51.667+0000 7fd77a17f7c0 -1 Falling back to public interface 2024-04-25T07:43:51.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:52.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:52.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:52.712 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:43:52.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:43:52.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:52.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:53.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:54.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:54.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:54.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:43:54.135 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:43:54.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:54.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:54.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:55.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:55.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:55.566 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:43:55.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:43:55.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:55.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:56.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:56.559 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:43:56.559+0000 7fd77a17f7c0 -1 osd.0 10 log_to_monitors true 2024-04-25T07:43:57.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:57.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:57.045 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:43:57.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:43:57.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:57.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:58.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:59.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:59.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:59.038 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:43:59.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:43:59.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:59.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:59.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:00.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:00.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:00.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:44:00.989 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:44:00.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:00.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:01.555 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/2709641501,v1:127.0.0.1:6803/2709641501] [v2:127.0.0.1:6804/2709641501,v1:127.0.0.1:6805/2709641501] exists,up eb9c7d09-7497-47e7-9feb-d104b6518f3f 2024-04-25T07:44:01.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:44:01.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:44:01.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:44:01.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:44:01.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: osd_pid0=153054 2024-04-25T07:44:01.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:463: TEST_bluestore_expand: wait_for_clean 2024-04-25T07:44:01.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:44:01.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:44:01.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:44:01.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:44:01.558 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:44:01.559 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:44:01.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:44:01.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:44:01.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:44:01.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:44:01.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:44:01.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:44:01.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:44:01.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:44:01.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:44:02.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T07:44:02.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:44:02.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:44:02.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:44:02.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574852 2024-04-25T07:44:02.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574852 2024-04-25T07:44:02.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-55834574852' 2024-04-25T07:44:02.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:44:02.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-55834574852 2024-04-25T07:44:02.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:44:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:44:02.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-55834574852 2024-04-25T07:44:02.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:44:02.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574852 2024-04-25T07:44:02.281 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 55834574852 2024-04-25T07:44:02.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-04-25T07:44:02.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:44:02.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 55834574852 2024-04-25T07:44:02.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:44:03.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:44:03.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:44:04.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574852 -lt 55834574852 2024-04-25T07:44:04.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:44:04.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:44:04.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:44:04.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:44:04.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:44:04.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:44:04.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:44:04.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:44:04.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:44:04.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:44:04.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:44:05.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:44:05.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:44:05.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:44:05.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:44:05.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:44:05.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:44:05.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:44:05.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-04-25T07:44:05.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-04-25T07:44:05.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: total_space_after=4294967296 2024-04-25T07:44:05.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-04-25T07:44:05.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-04-25T07:44:05.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep free 2024-04-25T07:44:05.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d: -f 2 2024-04-25T07:44:05.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d, -f 1 2024-04-25T07:44:05.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut '-d ' -f 2 2024-04-25T07:44:06.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: free_space_after=4280991744 2024-04-25T07:44:06.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:468: TEST_bluestore_expand: '[4294967296' '!=' '4294967296]' 2024-04-25T07:44:06.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 468: [4294967296: command not found 2024-04-25T07:44:06.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:474: TEST_bluestore_expand: total_space_added=0 2024-04-25T07:44:06.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:475: TEST_bluestore_expand: free_space_added=-45056 2024-04-25T07:44:06.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:477: TEST_bluestore_expand: let 'new_used_space=(0 - -45056)' 2024-04-25T07:44:06.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:478: TEST_bluestore_expand: echo 45056 2024-04-25T07:44:06.081 INFO:tasks.workunit.client.0.smithi102.stdout:45056 2024-04-25T07:44:06.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:480: TEST_bluestore_expand: '[' 45056 -gt 131072 ']' 2024-04-25T07:44:06.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 153054 2024-04-25T07:44:06.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: sleep 1 2024-04-25T07:44:07.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 153054 2024-04-25T07:44:07.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 487: kill: (153054) - No such process 2024-04-25T07:44:07.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:488: TEST_bluestore_expand: ceph osd down 0 2024-04-25T07:44:08.027 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T07:44:08.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:490: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-04-25T07:44:08.082 INFO:tasks.workunit.client.0.smithi102.stdout:qfsck bluestore.quick-fsck 2024-04-25T07:44:10.616 INFO:tasks.workunit.client.0.smithi102.stdout:qfsck success 2024-04-25T07:44:10.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:44:10.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:44:10.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:44:10.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:44:10.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:10.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:10.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:10.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:10.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:10.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:10.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:10.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:10.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:10.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:10.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:10.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:10.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:10.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:10.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:44:10.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:44:10.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:10.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:10.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:44:10.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95853 2024-04-25T07:44:10.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:10.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:10.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:44:10.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-bluefs-volume-ops 0 2024-04-25T07:44:10.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:44:10.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:44:10.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:44:10.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:10.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:10.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:10.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:10.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:10.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:10.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:10.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:10.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:10.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:10.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:10.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:10.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:10.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:10.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:10.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:10.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:10.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:44:10.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:44:10.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:10.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:10.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95853 2024-04-25T07:44:10.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95853 2024-04-25T07:44:10.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:10.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:10.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:44:10.830 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:44:10.831 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:44:10.883 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2024-04-25T07:44:10.883 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2024-04-25T07:44:10.935 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:44:10.940 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-config 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2024-04-25T07:44:10.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2024-04-25T07:44:10.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2024-04-25T07:44:10.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2024-04-25T07:44:10.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=dc1ce2e7-52ba-4200-8314-e721691f4838 --auth-supported=none ' 2024-04-25T07:44:10.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-04-25T07:44:10.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2024-04-25T07:44:10.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:44:10.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2024-04-25T07:44:10.944 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_config_track 2024-04-25T07:44:10.944 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_default_adjustment' 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:10.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:10.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:10.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:10.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:10.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:10.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:10.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:10.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:10.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:10.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:10.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:10.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:10.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:10.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:44:10.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:44:10.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:10.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:10.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:10.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154775 2024-04-25T07:44:10.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:10.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:10.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:44:10.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:44:10.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:10.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:10.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.154775 2024-04-25T07:44:10.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:44:10.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:44:10.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:44:10.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:44:10.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:44:10.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2024-04-25T07:44:10.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2024-04-25T07:44:10.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2024-04-25T07:44:10.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-04-25T07:44:10.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:44:10.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:44:10.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:44:10.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-04-25T07:44:10.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-04-25T07:44:11.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:44:11.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:11.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:11.491 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:11.492 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:11.492 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:11.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154775/$cluster-$name.asok' 2024-04-25T07:44:11.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154775/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:44:11.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:44:11.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:44:11.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:44:11.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:44:11.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:44:11.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:44:11.721 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:44:11.721 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:44:11.721 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:44:11.721 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:44:11.721 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:11.721 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:11.721 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154775/ceph-mon.a.asok 2024-04-25T07:44:11.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:44:11.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154775/ceph-mon.a.asok config get fsid 2024-04-25T07:44:11.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:44:11.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:44:11.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:44:11.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:44:11.840 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:44:11.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:44:11.841 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:44:11.841 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:44:11.841 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:44:11.841 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:11.841 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:11.841 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154775/ceph-mon.a.asok 2024-04-25T07:44:11.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:44:11.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154775/ceph-mon.a.asok config get mon_host 2024-04-25T07:44:11.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2024-04-25T07:44:11.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-04-25T07:44:11.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:44:11.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:44:11.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:44:11.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-04-25T07:44:11.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:44:12.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:44:12.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:12.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:12.852 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:12.852 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:12.852 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:12.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154775/$cluster-$name.asok' 2024-04-25T07:44:12.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:44:12.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154775/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:44:12.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2024-04-25T07:44:12.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dc1ce2e7-52ba-4200-8314-e721691f4838 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:44:12.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-04-25T07:44:12.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:44:12.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:12.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:12.888 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:12.888 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:12.888 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:12.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154775/$cluster-$name.asok' 2024-04-25T07:44:12.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.154775/$cluster-$name.asok' 2024-04-25T07:44:12.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:44:12.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:44:12.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:44:12.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-04-25T07:44:12.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-04-25T07:44:12.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:44:12.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:44:12.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:44:12.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:44:12.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:44:12.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2024-04-25T07:44:12.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-04-25T07:44:12.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:44:12.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b1a167f9-d9b7-48ba-a9d8-679b873d5187 2024-04-25T07:44:12.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b1a167f9-d9b7-48ba-a9d8-679b873d5187' 2024-04-25T07:44:12.890 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 b1a167f9-d9b7-48ba-a9d8-679b873d5187 2024-04-25T07:44:12.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:44:12.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMCSpm+lIUNhAAcMEXdSn1lrGo1r6PHg+LJA== 2024-04-25T07:44:12.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMCSpm+lIUNhAAcMEXdSn1lrGo1r6PHg+LJA=="}' 2024-04-25T07:44:12.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b1a167f9-d9b7-48ba-a9d8-679b873d5187 -i td/osd-config/0/new.json 2024-04-25T07:44:13.223 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:44:13.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-04-25T07:44:13.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dc1ce2e7-52ba-4200-8314-e721691f4838 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154775/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQDMCSpm+lIUNhAAcMEXdSn1lrGo1r6PHg+LJA== --osd-uuid b1a167f9-d9b7-48ba-a9d8-679b873d5187 2024-04-25T07:44:13.271 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:13.270+0000 7f3984bc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:13.290 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:13.290+0000 7f3984bc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:13.292 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:13.290+0000 7f3984bc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:13.292 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:13.294+0000 7f3984bc77c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:44:13.292 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:13.294+0000 7f3984bc77c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:44:13.292 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:13.294+0000 7f3984bc77c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:44:13.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:13.322+0000 7f3984bc77c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-04-25T07:44:19.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-04-25T07:44:19.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:44:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:44:19.853 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:44:19.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:44:20.382 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:44:20.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:44:20.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:44:20.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dc1ce2e7-52ba-4200-8314-e721691f4838 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154775/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-04-25T07:44:20.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:44:20.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:44:20.413 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:20.414+0000 7f980ccce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:20.415 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:20.414+0000 7f980ccce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:20.416 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:20.418+0000 7f980ccce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:20.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:44:20.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:44:20.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:44:20.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:44:20.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:44:20.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:20.840 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:44:20.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:44:20.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:20.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:20.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:20.982+0000 7f980ccce7c0 -1 Falling back to public interface 2024-04-25T07:44:21.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:22.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:22.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:22.232 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:44:22.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:44:22.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:22.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:22.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:23.622+0000 7f980ccce7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:44:23.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:23.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:23.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:44:23.632 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:44:23.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:23.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:24.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:25.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:25.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:25.135 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:44:25.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:44:25.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:25.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:25.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:26.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:26.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:26.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:44:26.664 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:44:26.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:26.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:27.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:28.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:28.135 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:44:28.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:44:28.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:28.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:28.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:29.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:29.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:29.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:44:29.564 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:44:29.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:29.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:30.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:31.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:31.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:31.029 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:44:31.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:44:31.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:31.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:31.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:32.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:32.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:32.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:44:32.537 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:44:32.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:32.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:32.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:33.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:33.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:33.963 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:44:33.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:44:33.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:33.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:34.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:35.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:35.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:35.487 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:44:35.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:44:35.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:35.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:35.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:36.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:36.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:36.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:44:36.978 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:44:36.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:36.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:37.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:38.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:38.392 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:44:38.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:38.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:44:38.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:38.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:39.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:40.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:40.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:40.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:44:40.207 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:44:40.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:40.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:41.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:42.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:42.401 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:44:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:44:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:43.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:44.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:44.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:44.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:44:44.296 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:44:44.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:44.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:45.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:46.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:46.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:46.625 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:44:46.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:44:46.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:46.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:47.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:48.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:48.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:48.496 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:44:48.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:44:48.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:48.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:49.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:50.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:50.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:50.051 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:44:50.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:44:50.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:50.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:50.522 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2011353384,v1:127.0.0.1:6803/2011353384] [v2:127.0.0.1:6804/2011353384,v1:127.0.0.1:6805/2011353384] exists,up b1a167f9-d9b7-48ba-a9d8-679b873d5187 2024-04-25T07:44:50.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:44:50.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:44:50.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:44:50.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2024-04-25T07:44:50.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:44:50.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:44:50.524 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:44:50.524 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:50.524 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:50.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154775/ceph-osd.0.asok 2024-04-25T07:44:50.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2024-04-25T07:44:50.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.154775/ceph-osd.0.asok log flush 2024-04-25T07:44:50.633 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T07:44:50.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:44:50.648 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:44:23.954+0000 7f980ccce7c0 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:44:50.648 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:44:26.622+0000 7f97fcc70640 10 log_client will send 2024-04-25T07:44:23.958518+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:44:50.648 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:44:26.846+0000 7f97fdc72640 10 log_client logged 2024-04-25T07:44:23.958518+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:44:50.648 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:44:48.889+0000 7f9808c88640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:44:50.648 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:44:49.625+0000 7f97fcc70640 10 log_client will send 2024-04-25T07:44:48.892947+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:44:50.648 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:44:49.733+0000 7f97fdc72640 10 log_client logged 2024-04-25T07:44:48.892947+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:44:50.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:44:50.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:44:50.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:44:50.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:44:50.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:50.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:50.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:50.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:50.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:50.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:50.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:50.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:50.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:50.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:50.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:50.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:50.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:50.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:50.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:50.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:44:50.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:44:50.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:50.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:50.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:50.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154775 2024-04-25T07:44:50.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:50.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:50.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:44:50.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:44:50.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:44:50.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:44:50.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:44:50.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:44:50.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:44:50.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:50.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:50.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:50.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:50.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:50.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:50.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:50.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:50.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:50.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:50.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:50.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:50.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:50.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:50.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:50.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:50.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:44:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:44:50.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:50.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:50.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:50.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154775 2024-04-25T07:44:50.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:50.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:50.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:44:50.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:44:50.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:50.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:50.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.154775 2024-04-25T07:44:50.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-04-25T07:44:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-04-25T07:44:51.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:44:51.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:51.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:51.480 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:51.480 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:51.480 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:51.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154775/$cluster-$name.asok' 2024-04-25T07:44:51.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154775/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:44:51.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:44:51.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:44:51.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:44:51.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:44:51.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:44:51.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:44:51.682 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:44:51.682 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:44:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:44:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:44:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:51.684 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154775/ceph-mon.a.asok 2024-04-25T07:44:51.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:44:51.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154775/ceph-mon.a.asok config get fsid 2024-04-25T07:44:51.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:44:51.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:44:51.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:44:51.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:44:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:44:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:44:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:44:51.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:44:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:44:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154775/ceph-mon.a.asok 2024-04-25T07:44:51.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:44:51.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154775/ceph-mon.a.asok config get mon_host 2024-04-25T07:44:51.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2024-04-25T07:44:51.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-04-25T07:44:51.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:44:51.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:44:51.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:44:51.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-04-25T07:44:51.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:44:52.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:44:52.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:52.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:52.806 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:52.806 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:52.806 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:52.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154775/$cluster-$name.asok' 2024-04-25T07:44:52.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:44:52.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154775/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:44:52.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2024-04-25T07:44:52.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-04-25T07:44:52.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=dc1ce2e7-52ba-4200-8314-e721691f4838 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:52.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154775/$cluster-$name.asok' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.154775/$cluster-$name.asok' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:44:52.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:44:52.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-04-25T07:44:52.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:44:52.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=410b02d8-f569-4bc8-bfc1-e7a8bb9c2dd0 2024-04-25T07:44:52.845 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 410b02d8-f569-4bc8-bfc1-e7a8bb9c2dd0 2024-04-25T07:44:52.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 410b02d8-f569-4bc8-bfc1-e7a8bb9c2dd0' 2024-04-25T07:44:52.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:44:52.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0CSpmGeJlMxAAtMjINWwALfV27BQp0T5y3A== 2024-04-25T07:44:52.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0CSpmGeJlMxAAtMjINWwALfV27BQp0T5y3A=="}' 2024-04-25T07:44:52.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 410b02d8-f569-4bc8-bfc1-e7a8bb9c2dd0 -i td/osd-config/0/new.json 2024-04-25T07:44:53.135 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:44:53.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-04-25T07:44:53.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=dc1ce2e7-52ba-4200-8314-e721691f4838 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154775/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0CSpmGeJlMxAAtMjINWwALfV27BQp0T5y3A== --osd-uuid 410b02d8-f569-4bc8-bfc1-e7a8bb9c2dd0 2024-04-25T07:44:53.186 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:53.185+0000 7fb6471137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:53.205 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:53.205+0000 7fb6471137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:53.206 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:53.205+0000 7fb6471137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:53.207 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:53.205+0000 7fb6471137c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:44:53.207 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:53.205+0000 7fb6471137c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:44:53.207 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:53.205+0000 7fb6471137c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:44:53.244 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:44:53.241+0000 7fb6471137c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-04-25T07:44:59.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-04-25T07:44:59.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:44:59.651 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:44:59.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:44:59.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:45:00.187 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:45:00.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:45:00.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=dc1ce2e7-52ba-4200-8314-e721691f4838 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154775/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:45:00.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:45:00.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:45:00.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:45:00.217 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:00.217+0000 7f272db067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:00.219 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:00.217+0000 7f272db067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:00.220 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:00.221+0000 7f272db067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:00.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:45:00.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:45:00.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:45:00.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:45:00.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:45:00.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:00.638 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:45:00.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:45:00.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:00.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:00.797 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:00.797+0000 7f272db067c0 -1 Falling back to public interface 2024-04-25T07:45:01.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:02.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:02.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:02.049 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:45:02.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:45:02.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:02.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:02.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:03.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:03.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:03.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:45:03.455 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:45:03.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:03.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:03.476 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:03.477+0000 7f272db067c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:45:04.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:05.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:05.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:05.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:45:05.037 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:45:05.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:05.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:05.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:06.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:06.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:06.488 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:45:06.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:45:06.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:06.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:07.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:08.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:08.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:08.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:45:08.053 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:45:08.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:08.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:08.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:09.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:09.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:09.605 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:45:09.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:45:09.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:09.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:10.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:11.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:11.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:11.069 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:45:11.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:45:11.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:11.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:11.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:12.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:12.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:12.713 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:45:12.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:45:12.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:12.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:13.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:14.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:14.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:14.126 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:45:14.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:45:14.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:14.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:14.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:15.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:15.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:15.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:45:15.735 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:45:15.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:15.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:16.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:17.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:17.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:17.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:45:17.198 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:45:17.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:17.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:18.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:19.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:19.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:19.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:45:19.348 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:45:19.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:19.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:19.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:20.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:20.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:20.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:45:20.873 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:45:20.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:20.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:21.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:22.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:22.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:22.662 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:45:22.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:45:22.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:22.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:23.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:24.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:24.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:24.511 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:45:24.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:45:24.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:24.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:25.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:26.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:26.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:26.370 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:45:26.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:45:26.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:26.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:27.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:28.206 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:45:28.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:45:28.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:28.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:28.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:29.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:29.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:29.794 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:45:29.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:45:29.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:29.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:30.229 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3350964367,v1:127.0.0.1:6803/3350964367] [v2:127.0.0.1:6804/3350964367,v1:127.0.0.1:6805/3350964367] exists,up 410b02d8-f569-4bc8-bfc1-e7a8bb9c2dd0 2024-04-25T07:45:30.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:45:30.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:45:30.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:45:30.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2024-04-25T07:45:30.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2024-04-25T07:45:30.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2024-04-25T07:45:30.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2024-04-25T07:45:30.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2024-04-25T07:45:30.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2024-04-25T07:45:30.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:45:30.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2024-04-25T07:45:30.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2024-04-25T07:45:30.397 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T07:45:30.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2024-04-25T07:45:30.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:45:30.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:45:30.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:45:30.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:30.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:45:30.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154775/ceph-osd.0.asok 2024-04-25T07:45:30.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2024-04-25T07:45:30.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.154775/ceph-osd.0.asok log flush 2024-04-25T07:45:30.518 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T07:45:30.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:45:30.533 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:45:30.384+0000 7f272c2c5640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2024-04-25T07:45:30.534 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T07:45:30.476+0000 7f271daa8640 10 log_client will send 2024-04-25T07:45:30.389438+0000 osd.0 (osd.0) 3 : cluster [WRN] osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2024-04-25T07:45:30.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2024-04-25T07:45:30.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2024-04-25T07:45:30.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:45:30.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:45:30.536 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:45:30.536 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:30.536 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:45:30.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154775/ceph-osd.0.asok 2024-04-25T07:45:30.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2024-04-25T07:45:30.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.154775/ceph-osd.0.asok log reopen 2024-04-25T07:45:30.648 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T07:45:30.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:45:30.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:45:30.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:45:30.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:45:30.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:45:30.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:45:30.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:45:30.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:45:30.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:45:30.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:45:30.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:45:30.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:45:30.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:45:30.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:45:30.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:45:30.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:45:30.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:30.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:45:30.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:45:30.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:30.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:45:30.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:45:30.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:45:30.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:45:30.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:30.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:45:30.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154775 2024-04-25T07:45:30.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:45:30.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:45:30.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:45:30.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:45:30.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:45:30.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:45:30.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:45:30.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:45:30.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:45:30.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:45:30.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:45:30.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:45:30.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:45:30.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:45:30.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:45:30.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:45:30.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:45:30.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:45:30.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:45:30.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:45:30.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:45:30.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:30.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:45:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:45:30.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:30.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:45:30.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:45:30.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:45:30.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:45:30.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:30.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:45:30.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154775 2024-04-25T07:45:30.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:45:30.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:45:30.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:45:30.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:45:30.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:30.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:45:30.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.154775 2024-04-25T07:45:30.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:45:30.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:45:30.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:45:30.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:45:30.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2024-04-25T07:45:30.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2024-04-25T07:45:30.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2024-04-25T07:45:30.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2024-04-25T07:45:30.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2024-04-25T07:45:30.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2024-04-25T07:45:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2024-04-25T07:45:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2024-04-25T07:45:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2024-04-25T07:45:30.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2024-04-25T07:45:30.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2024-04-25T07:45:30.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2024-04-25T07:45:30.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2024-04-25T07:45:30.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2024-04-25T07:45:30.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2024-04-25T07:45:30.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2024-04-25T07:45:30.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2024-04-25T07:45:30.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2024-04-25T07:45:30.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:45:30.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:45:30.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:45:30.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:45:30.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:45:30.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:30.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:45:30.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:45:30.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:30.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:45:30.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:45:30.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:45:30.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:45:30.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:30.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:45:30.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154775 2024-04-25T07:45:30.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:45:30.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:45:30.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:45:30.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-config 0 2024-04-25T07:45:30.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:45:30.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:45:30.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:45:30.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:45:30.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:45:30.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:45:30.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:45:30.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:45:30.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:45:30.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:45:30.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:45:30.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:45:30.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:45:30.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:30.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:45:30.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:45:30.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:45:30.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:45:30.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:30.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154775 2024-04-25T07:45:30.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154775 2024-04-25T07:45:30.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:45:30.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:45:30.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:45:30.928 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:45:30.929 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:45:30.982 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2024-04-25T07:45:30.982 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2024-04-25T07:45:31.038 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:45:31.043 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:45:31.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:45:31.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:45:31.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-copy-from 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2024-04-25T07:45:31.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2024-04-25T07:45:31.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=ced8e6fe-ae1f-493d-9338-64454a938f19 --auth-supported=none ' 2024-04-25T07:45:31.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2024-04-25T07:45:31.045 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2024-04-25T07:45:31.045 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:45:31.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2024-04-25T07:45:31.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2024-04-25T07:45:31.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2024-04-25T07:45:31.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2024-04-25T07:45:31.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2024-04-25T07:45:31.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:45:31.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:45:31.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:45:31.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:45:31.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:45:31.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:45:31.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:45:31.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:45:31.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:45:31.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:45:31.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:45:31.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:45:31.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:45:31.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:45:31.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:45:31.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:31.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:45:31.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:45:31.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:31.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:45:31.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:45:31.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:45:31.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:45:31.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:31.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158943 2024-04-25T07:45:31.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.158943 2024-04-25T07:45:31.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:45:31.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:45:31.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2024-04-25T07:45:31.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:45:31.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:31.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158943 2024-04-25T07:45:31.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.158943 2024-04-25T07:45:31.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:45:31.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:45:31.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:45:31.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:45:31.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2024-04-25T07:45:31.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2024-04-25T07:45:31.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2024-04-25T07:45:31.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2024-04-25T07:45:31.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2024-04-25T07:45:31.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:45:31.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:45:31.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:45:31.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2024-04-25T07:45:31.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2024-04-25T07:45:31.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:45:31.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:45:31.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:45:31.561 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:45:31.561 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:31.562 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158943 2024-04-25T07:45:31.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158943/$cluster-$name.asok' 2024-04-25T07:45:31.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.158943/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:45:31.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:45:31.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:45:31.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:45:31.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:45:31.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:45:31.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:45:31.783 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:45:31.783 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:45:31.783 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:45:31.783 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:45:31.783 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:31.784 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158943 2024-04-25T07:45:31.784 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.158943/ceph-mon.a.asok 2024-04-25T07:45:31.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:45:31.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.158943/ceph-mon.a.asok config get fsid 2024-04-25T07:45:31.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:45:31.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:45:31.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:45:31.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:45:31.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:45:31.905 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:45:31.905 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:45:31.905 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:45:31.905 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:45:31.906 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:31.906 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158943 2024-04-25T07:45:31.906 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.158943/ceph-mon.a.asok 2024-04-25T07:45:31.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:45:31.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.158943/ceph-mon.a.asok config get mon_host 2024-04-25T07:45:32.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2024-04-25T07:45:32.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2024-04-25T07:45:32.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:45:32.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:45:32.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:45:32.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2024-04-25T07:45:32.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:45:32.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:45:32.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:45:32.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:45:32.939 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:45:32.939 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:32.939 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158943 2024-04-25T07:45:32.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158943/$cluster-$name.asok' 2024-04-25T07:45:32.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:45:32.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.158943/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:45:32.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ced8e6fe-ae1f-493d-9338-64454a938f19 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:45:32.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158943 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158943/$cluster-$name.asok' 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.158943/$cluster-$name.asok' 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:45:32.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:45:32.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:45:32.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:45:32.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:45:32.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:45:32.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/0 2024-04-25T07:45:32.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:45:32.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f9aebe16-6906-47a4-9a04-030ccc4262c8 2024-04-25T07:45:32.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f9aebe16-6906-47a4-9a04-030ccc4262c8' 2024-04-25T07:45:32.979 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 f9aebe16-6906-47a4-9a04-030ccc4262c8 2024-04-25T07:45:32.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:45:32.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcCipm7MFnOxAAD8FUySVjR+LNbUoCaMkIIg== 2024-04-25T07:45:32.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcCipm7MFnOxAAD8FUySVjR+LNbUoCaMkIIg=="}' 2024-04-25T07:45:32.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f9aebe16-6906-47a4-9a04-030ccc4262c8 -i td/osd-copy-from/0/new.json 2024-04-25T07:45:33.317 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:45:33.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/0/new.json 2024-04-25T07:45:33.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ced8e6fe-ae1f-493d-9338-64454a938f19 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158943/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAcCipm7MFnOxAAD8FUySVjR+LNbUoCaMkIIg== --osd-uuid f9aebe16-6906-47a4-9a04-030ccc4262c8 2024-04-25T07:45:33.361 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:33.360+0000 7f71d6d187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:33.394 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:33.392+0000 7f71d6d187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:33.396 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:33.396+0000 7f71d6d187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:33.397 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:33.396+0000 7f71d6d187c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:45:33.397 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:33.396+0000 7f71d6d187c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:45:33.397 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:33.396+0000 7f71d6d187c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:45:33.434 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:33.432+0000 7f71d6d187c0 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2024-04-25T07:45:39.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/0/keyring 2024-04-25T07:45:39.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:45:39.975 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:45:39.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:45:39.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:45:40.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:45:40.563 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:45:40.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:45:40.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ced8e6fe-ae1f-493d-9338-64454a938f19 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158943/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:45:40.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:45:40.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:45:40.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:40.588+0000 7ff35af147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:40.594 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:40.592+0000 7ff35af147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:40.595 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:40.592+0000 7ff35af147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:40.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:45:40.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:45:40.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:45:40.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:45:40.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:45:40.978 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:45:40.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:40.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:45:40.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:40.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:41.161 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:41.160+0000 7ff35af147c0 -1 Falling back to public interface 2024-04-25T07:45:41.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:42.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:42.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:42.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:45:42.418 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:45:42.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:42.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:42.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:43.760 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:45:43.756+0000 7ff35af147c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:45:43.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:43.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:43.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:45:43.818 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:45:43.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:43.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:44.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:45.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:45.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:45.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:45:45.514 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:45:45.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:45.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:45.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:46.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:46.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:46.938 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:45:46.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:45:46.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:46.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:47.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:48.372 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:45:48.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:45:48.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:48.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:48.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:49.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:49.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:49.894 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:45:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:45:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:51.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:51.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:51.317 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:45:51.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:45:51.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:51.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:51.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:52.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:52.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:52.832 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:45:52.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:45:52.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:52.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:53.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:54.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:54.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:54.260 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:45:54.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:45:54.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:54.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:54.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:55.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:55.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:55.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:45:55.810 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:45:55.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:55.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:56.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:57.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:57.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:57.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:45:57.294 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:45:57.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:57.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:57.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:58.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:58.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:58.806 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:45:58.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:45:58.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:58.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:59.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:00.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:00.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:00.647 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:46:00.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:46:00.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:00.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:01.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:02.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:02.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:02.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:03.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:04.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:04.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:04.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:46:04.631 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:46:04.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:04.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:05.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:06.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:06.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:06.470 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:46:06.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:46:06.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:06.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:07.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:08.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:08.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:08.466 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:46:08.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:46:08.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:08.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:09.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:10.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:10.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:10.321 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:46:10.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:46:10.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:10.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:10.736 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2766100214,v1:127.0.0.1:6803/2766100214] [v2:127.0.0.1:6804/2766100214,v1:127.0.0.1:6805/2766100214] exists,up f9aebe16-6906-47a4-9a04-030ccc4262c8 2024-04-25T07:46:10.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ced8e6fe-ae1f-493d-9338-64454a938f19 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2024-04-25T07:46:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158943 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158943/$cluster-$name.asok' 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.158943/$cluster-$name.asok' 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-04-25T07:46:10.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-04-25T07:46:10.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:46:10.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:46:10.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:46:10.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:46:10.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:46:10.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:46:10.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/1 2024-04-25T07:46:10.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:46:10.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c7dd453-83e0-4f68-8fb7-3116b492025d 2024-04-25T07:46:10.743 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 0c7dd453-83e0-4f68-8fb7-3116b492025d 2024-04-25T07:46:10.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0c7dd453-83e0-4f68-8fb7-3116b492025d' 2024-04-25T07:46:10.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:46:10.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCCipmXHhyLRAASwxIlmjYJFyoPB5wmI7mBg== 2024-04-25T07:46:10.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCCipmXHhyLRAASwxIlmjYJFyoPB5wmI7mBg=="}' 2024-04-25T07:46:10.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c7dd453-83e0-4f68-8fb7-3116b492025d -i td/osd-copy-from/1/new.json 2024-04-25T07:46:11.243 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:46:11.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/1/new.json 2024-04-25T07:46:11.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=ced8e6fe-ae1f-493d-9338-64454a938f19 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158943/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCCipmXHhyLRAASwxIlmjYJFyoPB5wmI7mBg== --osd-uuid 0c7dd453-83e0-4f68-8fb7-3116b492025d 2024-04-25T07:46:11.291 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:11.291+0000 7f3e546a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:11.338 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:11.339+0000 7f3e546a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:11.340 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:11.339+0000 7f3e546a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:11.340 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:11.343+0000 7f3e546a67c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:46:11.341 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:11.343+0000 7f3e546a67c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:46:11.341 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:11.343+0000 7f3e546a67c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:46:11.376 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:11.375+0000 7f3e546a67c0 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2024-04-25T07:46:18.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/1/keyring 2024-04-25T07:46:18.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:46:18.417 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:46:18.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:46:18.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:46:18.969 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:46:18.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:46:18.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=ced8e6fe-ae1f-493d-9338-64454a938f19 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158943/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:46:18.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:46:18.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:46:18.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:46:18.992 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:18.991+0000 7f05618977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:18.994 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:18.995+0000 7f05618977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:18.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:18.995+0000 7f05618977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:19.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:46:19.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:46:19.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:46:19.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:46:19.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:46:19.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:19.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:46:19.426 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:46:19.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:19.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:19.570 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:19.571+0000 7f05618977c0 -1 Falling back to public interface 2024-04-25T07:46:19.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:20.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:20.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:20.865 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:46:20.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:46:20.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:20.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:21.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:22.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:22.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:22.276 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:46:22.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:46:22.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:22.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:22.343 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:22.343+0000 7f05618977c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:46:22.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:23.720 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:23.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:24.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:25.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:25.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:25.291 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:46:25.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:46:25.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:25.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:25.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:26.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:26.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:26.784 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:46:26.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:46:26.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:26.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:27.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:28.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:28.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:28.337 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:46:28.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:46:28.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:28.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:28.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:29.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:29.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:29.833 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:46:29.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:46:29.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:29.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:30.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:31.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:31.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:31.459 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:46:31.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:46:31.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:31.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:31.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:32.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:32.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:32.890 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:46:32.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:46:32.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:32.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:33.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:34.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:34.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:34.325 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:46:34.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:46:34.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:34.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:34.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:35.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:35.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:35.781 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:46:35.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:46:35.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:35.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:36.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:37.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:37.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:37.261 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:46:37.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:46:37.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:37.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:38.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:39.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:39.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:39.091 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:46:39.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:46:39.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:39.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:40.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:41.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:41.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:41.377 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:46:41.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:46:41.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:41.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:42.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:43.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:43.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:43.247 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:46:43.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:46:43.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:43.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:44.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:45.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:45.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:45.015 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:46:45.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:46:45.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:45.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:46.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:47.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:47.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:47.317 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:46:47.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:46:47.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:47.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:48.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:49.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:49.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:49.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:46:49.376 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:46:49.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:49.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:46:49.800 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/559075999,v1:127.0.0.1:6811/559075999] [v2:127.0.0.1:6812/559075999,v1:127.0.0.1:6813/559075999] exists,up 0c7dd453-83e0-4f68-8fb7-3116b492025d 2024-04-25T07:46:49.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:46:49.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:46:49.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:46:49.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2024-04-25T07:46:49.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:46:50.213 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-04-25T07:46:50.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:46:50.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:46:50.748 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' already exists 2024-04-25T07:46:50.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:46:51.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:46:54.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2024-04-25T07:46:54.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2024-04-25T07:46:54.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2024-04-25T07:46:54.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2024-04-25T07:46:54.567 INFO:tasks.workunit.client.0.smithi102.stdout:rbd/foo2 mtime 2024-04-25T07:46:54.000000+0000, size 572320 2024-04-25T07:46:54.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2024-04-25T07:46:54.707 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T07:46:54.735 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T07:46:54.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2024-04-25T07:46:54.786 INFO:tasks.workunit.client.0.smithi102.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2024-04-25T07:46:54.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2024-04-25T07:46:54.831 INFO:tasks.workunit.client.0.smithi102.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2024-04-25T07:46:54.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2024-04-25T07:46:54.970 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T07:46:54.998 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T07:46:55.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2024-04-25T07:46:55.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2024-04-25T07:46:55.313 INFO:tasks.workunit.client.0.smithi102.stdout:rbd/foo3 mtime 2024-04-25T07:46:54.000000+0000, size 572320 2024-04-25T07:46:55.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2024-04-25T07:46:55.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:46:55.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:46:55.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:46:55.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:55.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:55.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:55.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:55.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:55.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:55.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:55.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:55.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:55.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:55.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:55.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:55.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:55.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:55.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:55.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:55.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:55.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:46:55.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:46:55.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:55.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:55.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158943 2024-04-25T07:46:55.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.158943 2024-04-25T07:46:55.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:55.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:55.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:46:55.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-copy-from 0 2024-04-25T07:46:55.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:46:55.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:46:55.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:46:55.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:55.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:55.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:55.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:55.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:55.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:55.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:55.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:55.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:55.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:55.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:55.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:55.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:55.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:55.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:55.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:55.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:46:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:46:55.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:55.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:55.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158943 2024-04-25T07:46:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.158943 2024-04-25T07:46:55.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:55.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:55.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:46:55.664 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:46:55.665 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:46:55.717 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2024-04-25T07:46:55.718 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2024-04-25T07:46:55.778 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:46:55.785 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:46:55.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:46:55.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:46:55.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:46:55.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:46:55.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:46:55.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:46:55.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:46:55.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:46:55.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-dup 2024-04-25T07:46:55.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2024-04-25T07:46:55.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2024-04-25T07:46:55.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:46:55.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2024-04-25T07:46:55.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2024-04-25T07:46:55.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2024-04-25T07:46:55.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=cef62c35-4bb0-4ce5-8007-d4852895eefc --auth-supported=none ' 2024-04-25T07:46:55.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:46:55.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2024-04-25T07:46:55.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-04-25T07:46:55.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2024-04-25T07:46:55.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:46:55.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs= 2024-04-25T07:46:55.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:46:55.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-dup 0 2024-04-25T07:46:55.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2024-04-25T07:46:55.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:46:55.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2024-04-25T07:46:55.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:55.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:55.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:55.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:55.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:55.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:55.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:55.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:55.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:55.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:55.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:55.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:55.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:55.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:55.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:55.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:55.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:55.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:46:55.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2024-04-25T07:46:55.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:55.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:55.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163027 2024-04-25T07:46:55.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.163027 2024-04-25T07:46:55.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:55.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:55.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:46:55.800 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:46:55.801 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:46:55.857 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2024-04-25T07:46:55.857 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2024-04-25T07:46:55.913 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:46:55.918 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-fast-mark-down 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2024-04-25T07:46:55.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2024-04-25T07:46:55.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2024-04-25T07:46:55.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2024-04-25T07:46:55.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2024-04-25T07:46:55.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2024-04-25T07:46:55.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none ' 2024-04-25T07:46:55.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none ' 2024-04-25T07:46:55.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2024-04-25T07:46:55.921 INFO:tasks.workunit.client.0.smithi102.stdout:Ensuring old behavior is there... 2024-04-25T07:46:55.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:55.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:55.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:55.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:55.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:55.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:55.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:55.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:55.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:55.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:55.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:55.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:55.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:55.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:55.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:46:55.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:46:55.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:55.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:55.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:46:55.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.163056 2024-04-25T07:46:55.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:55.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:55.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-04-25T07:46:55.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:46:55.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:55.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:46:55.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.163056 2024-04-25T07:46:55.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:46:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:46:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:46:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:46:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-04-25T07:46:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-04-25T07:46:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-04-25T07:46:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:46:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:46:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:46:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-04-25T07:46:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-04-25T07:46:55.966 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:55.966+0000 7fc64691bc80 -1 failed for service _ceph-mon._tcp 2024-04-25T07:46:55.966 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:55.966+0000 7fc64691bc80 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-25T07:46:55.966 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T07:46:55.967 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:55.966+0000 7fc64691bc80 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2024-04-25T07:46:56.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:46:56.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:56.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:56.421 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:56.422 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:56.422 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:46:56.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.163056/$cluster-$name.asok' 2024-04-25T07:46:56.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T07:46:56.617 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:56.618+0000 7f923a3dec80 -1 failed for service _ceph-mon._tcp 2024-04-25T07:46:56.617 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:56.618+0000 7f923a3dec80 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-25T07:46:56.618 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T07:46:56.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:56.618+0000 7f923a3dec80 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2024-04-25T07:46:56.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2024-04-25T07:46:56.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2024-04-25T07:46:56.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-04-25T07:46:56.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-04-25T07:46:56.626 INFO:tasks.workunit.client.0.smithi102.stdout:Testing async msgr... 2024-04-25T07:46:56.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:46:56.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2024-04-25T07:46:56.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2024-04-25T07:46:56.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-04-25T07:46:56.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-04-25T07:46:56.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-04-25T07:46:56.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:46:56.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:46:56.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:46:56.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:56.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:56.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:56.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:56.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:56.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:56.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:56.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:56.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:56.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:56.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:56.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:56.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:56.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:56.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:56.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:56.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:56.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:46:56.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:46:56.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:56.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:56.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:46:56.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.163056 2024-04-25T07:46:56.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:56.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:56.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-04-25T07:46:56.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:46:56.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:56.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:46:56.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.163056 2024-04-25T07:46:56.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:46:56.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:46:56.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:46:56.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-04-25T07:46:56.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-04-25T07:46:56.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-04-25T07:46:56.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:46:56.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:46:56.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:46:56.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-04-25T07:46:56.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-04-25T07:46:57.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:46:57.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:57.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:57.142 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:57.142 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:57.142 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:46:57.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.163056/$cluster-$name.asok' 2024-04-25T07:46:57.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T07:46:57.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:46:57.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:46:57.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:46:57.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:46:57.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:46:57.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:46:57.363 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:46:57.363 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:46:57.363 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:46:57.364 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:46:57.364 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:57.364 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:46:57.364 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.163056/ceph-mon.a.asok 2024-04-25T07:46:57.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:46:57.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.163056/ceph-mon.a.asok config get fsid 2024-04-25T07:46:57.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:46:57.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:46:57.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:46:57.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:46:57.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:46:57.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:46:57.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:46:57.482 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:46:57.482 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:46:57.482 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:57.482 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:46:57.482 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.163056/ceph-mon.a.asok 2024-04-25T07:46:57.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:46:57.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.163056/ceph-mon.a.asok config get mon_host 2024-04-25T07:46:57.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2024-04-25T07:46:57.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2024-04-25T07:46:57.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:46:57.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:46:57.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:46:57.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2024-04-25T07:46:57.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:46:58.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:46:58.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:58.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:58.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:58.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:58.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:46:58.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.163056/$cluster-$name.asok' 2024-04-25T07:46:58.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:46:58.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:46:58.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:46:58.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2024-04-25T07:46:58.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:46:58.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:46:58.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:46:58.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.163056/$cluster-$name.asok' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:46:58.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:46:58.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:46:58.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:46:58.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/0 2024-04-25T07:46:58.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:46:58.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=236b28e9-0737-4f05-adf8-0e862d505c87 2024-04-25T07:46:58.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 236b28e9-0737-4f05-adf8-0e862d505c87' 2024-04-25T07:46:58.524 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 236b28e9-0737-4f05-adf8-0e862d505c87 2024-04-25T07:46:58.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:46:58.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQByCipmJ+ZWIBAAwCpoAFLqSuJXtXZo2SQqTA== 2024-04-25T07:46:58.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQByCipmJ+ZWIBAAwCpoAFLqSuJXtXZo2SQqTA=="}' 2024-04-25T07:46:58.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 236b28e9-0737-4f05-adf8-0e862d505c87 -i td/osd-fast-mark-down/0/new.json 2024-04-25T07:46:58.839 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:46:58.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/0/new.json 2024-04-25T07:46:58.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQByCipmJ+ZWIBAAwCpoAFLqSuJXtXZo2SQqTA== --osd-uuid 236b28e9-0737-4f05-adf8-0e862d505c87 2024-04-25T07:46:58.890 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:58.890+0000 7f5dc6d747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:58.907 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:58.906+0000 7f5dc6d747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:58.909 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:58.910+0000 7f5dc6d747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:58.910 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:58.910+0000 7f5dc6d747c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:46:58.910 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:58.910+0000 7f5dc6d747c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:46:58.910 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:58.910+0000 7f5dc6d747c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:46:58.948 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:46:58.946+0000 7f5dc6d747c0 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2024-04-25T07:47:05.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2024-04-25T07:47:05.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:47:05.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:47:05.548 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:47:05.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:47:06.093 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:47:06.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:47:06.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:47:06.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:47:06.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:47:06.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:47:06.126 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:06.126+0000 7f81bdb357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:06.128 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:06.126+0000 7f81bdb357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:06.129 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:06.130+0000 7f81bdb357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:06.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:47:06.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:47:06.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:47:06.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:47:06.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:47:06.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:06.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:47:06.523 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:47:06.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:06.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:06.690 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:06.690+0000 7f81bdb357c0 -1 Falling back to public interface 2024-04-25T07:47:06.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:07.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:07.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:07.931 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:47:07.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:47:07.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:07.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:08.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:09.275 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:09.274+0000 7f81bdb357c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:47:09.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:09.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:09.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:47:09.339 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:47:09.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:09.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:09.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:10.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:10.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:10.973 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:47:10.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:47:10.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:10.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:11.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:12.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:12.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:12.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:47:12.499 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:47:12.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:12.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:13.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:14.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:14.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:14.112 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:47:14.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:47:14.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:14.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:14.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:15.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:15.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:15.534 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:47:15.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:47:15.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:15.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:15.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:16.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:16.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:16.972 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:47:16.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:47:16.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:16.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:17.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:18.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:18.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:18.449 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:47:18.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:47:18.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:18.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:18.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:19.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:19.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:19.874 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:47:19.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:47:19.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:19.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:20.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:21.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:21.438 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:47:21.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:21.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:47:21.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:21.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:21.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:22.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:22.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:22.886 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:47:22.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:47:22.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:22.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:24.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:25.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:25.288 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:47:25.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:25.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:47:25.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:25.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:25.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:27.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:27.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:27.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:47:27.002 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:47:27.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:27.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:27.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:28.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:28.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:28.928 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:47:28.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:47:28.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:28.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:30.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:31.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:31.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:31.082 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:47:31.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:47:31.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:31.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:33.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:33.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:33.534 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:47:33.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:47:33.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:33.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:34.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:35.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:35.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:35.327 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:47:35.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:47:35.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:35.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:35.808 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2908118016,v1:127.0.0.1:6803/2908118016] [v2:127.0.0.1:6804/2908118016,v1:127.0.0.1:6805/2908118016] exists,up 236b28e9-0737-4f05-adf8-0e862d505c87 2024-04-25T07:47:35.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:47:35.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:47:35.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:47:35.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2024-04-25T07:47:35.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=164202 2024-04-25T07:47:35.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:47:35.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2024-04-25T07:47:35.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:47:35.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:47:35.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:47:35.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:47:35.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:47:35.814 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.163056/$cluster-$name.asok' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:47:35.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/1 2024-04-25T07:47:35.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:47:35.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=432d672d-3372-4642-a298-c6197584c634 2024-04-25T07:47:35.816 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 432d672d-3372-4642-a298-c6197584c634 2024-04-25T07:47:35.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 432d672d-3372-4642-a298-c6197584c634' 2024-04-25T07:47:35.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:47:35.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCXCipmi2AFMhAAD7aqhjLAV/2yE22Wci83Jg== 2024-04-25T07:47:35.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCXCipmi2AFMhAAD7aqhjLAV/2yE22Wci83Jg=="}' 2024-04-25T07:47:35.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 432d672d-3372-4642-a298-c6197584c634 -i td/osd-fast-mark-down/1/new.json 2024-04-25T07:47:36.294 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:47:36.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/1/new.json 2024-04-25T07:47:36.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCXCipmi2AFMhAAD7aqhjLAV/2yE22Wci83Jg== --osd-uuid 432d672d-3372-4642-a298-c6197584c634 2024-04-25T07:47:36.346 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:36.345+0000 7ff3f7a0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:36.356 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:36.353+0000 7ff3f7a0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:36.358 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:36.357+0000 7ff3f7a0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:36.358 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:36.357+0000 7ff3f7a0a7c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:47:36.358 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:36.357+0000 7ff3f7a0a7c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:47:36.358 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:36.357+0000 7ff3f7a0a7c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:47:36.396 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:36.393+0000 7ff3f7a0a7c0 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2024-04-25T07:47:43.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2024-04-25T07:47:43.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:47:43.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:47:43.364 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:47:43.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:47:43.924 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:47:43.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:47:43.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:47:43.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:47:43.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:47:43.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:47:43.957 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:43.957+0000 7fa3e1acc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:43.970 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:43.969+0000 7fa3e1acc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:43.971 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:43.969+0000 7fa3e1acc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:44.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:47:44.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:47:44.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:47:44.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:47:44.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:47:44.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:44.335 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:47:44.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:47:44.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:44.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:44.538 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:44.537+0000 7fa3e1acc7c0 -1 Falling back to public interface 2024-04-25T07:47:44.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:45.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:45.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:45.765 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:47:45.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:47:45.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:45.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:46.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:47.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:47.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:47.183 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:47:47.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:47:47.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:47.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:47.257 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:47:47.257+0000 7fa3e1acc7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:47:47.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:48.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:48.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:48.835 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:47:48.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:47:48.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:48.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:49.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:50.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:50.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:50.340 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:47:50.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:47:50.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:50.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:50.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:51.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:51.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:51.774 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:47:51.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:47:51.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:51.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:53.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:53.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:53.342 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:47:53.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:47:53.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:53.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:53.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:54.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:54.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:54.905 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:47:54.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:47:54.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:54.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:55.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:56.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:56.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:56.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:47:56.573 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:47:56.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:56.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:56.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:57.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:57.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:57.996 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:47:57.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:47:57.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:57.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:58.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:59.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:59.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:59.433 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:47:59.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:47:59.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:59.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:59.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:00.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:00.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:00.952 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:48:00.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:48:00.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:00.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:01.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:02.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:02.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:02.480 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:48:02.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:48:02.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:02.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:02.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:03.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:03.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:03.951 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:48:03.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:48:03.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:03.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:04.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:05.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:05.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:05.858 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:48:05.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:48:05.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:05.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:07.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:08.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:08.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:08.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:48:08.064 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:48:08.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:08.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:09.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:10.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:10.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:10.067 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:48:10.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:48:10.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:10.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:10.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:11.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:11.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:11.841 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:48:11.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:48:11.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:11.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:12.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:13.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:13.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:13.649 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:48:13.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:48:13.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:13.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:14.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:15.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:15.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:15.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:48:15.067 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T07:48:15.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:15.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:15.509 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1158808303,v1:127.0.0.1:6811/1158808303] [v2:127.0.0.1:6812/1158808303,v1:127.0.0.1:6813/1158808303] exists,up 432d672d-3372-4642-a298-c6197584c634 2024-04-25T07:48:15.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:48:15.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:48:15.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:48:15.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2024-04-25T07:48:15.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=165941 2024-04-25T07:48:15.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:48:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2024-04-25T07:48:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:48:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:48:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:48:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:48:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2024-04-25T07:48:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:48:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:48:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:48:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:48:15.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2024-04-25T07:48:15.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2024-04-25T07:48:15.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:48:15.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:48:15.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:48:15.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:48:15.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:15.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:15.516 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:15.516 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:15.516 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:48:15.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.163056/$cluster-$name.asok' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:48:15.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/2 2024-04-25T07:48:15.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:48:15.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b8663be-3b1f-4455-b65a-ce4b7e67661b 2024-04-25T07:48:15.518 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 4b8663be-3b1f-4455-b65a-ce4b7e67661b 2024-04-25T07:48:15.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4b8663be-3b1f-4455-b65a-ce4b7e67661b' 2024-04-25T07:48:15.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:48:15.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/CipmRNZiIBAAczLgUzlYdphmTyjV3HQ+JA== 2024-04-25T07:48:15.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/CipmRNZiIBAAczLgUzlYdphmTyjV3HQ+JA=="}' 2024-04-25T07:48:15.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b8663be-3b1f-4455-b65a-ce4b7e67661b -i td/osd-fast-mark-down/2/new.json 2024-04-25T07:48:16.038 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:48:16.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/2/new.json 2024-04-25T07:48:16.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC/CipmRNZiIBAAczLgUzlYdphmTyjV3HQ+JA== --osd-uuid 4b8663be-3b1f-4455-b65a-ce4b7e67661b 2024-04-25T07:48:16.091 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:16.088+0000 7fcc3aae67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:16.109 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:16.108+0000 7fcc3aae67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:16.110 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:16.108+0000 7fcc3aae67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:16.111 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:16.108+0000 7fcc3aae67c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:48:16.111 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:16.108+0000 7fcc3aae67c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:48:16.111 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:16.108+0000 7fcc3aae67c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:48:16.149 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:16.148+0000 7fcc3aae67c0 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2024-04-25T07:48:23.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2024-04-25T07:48:23.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:48:23.142 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:48:23.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:48:23.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:48:23.691 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:48:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:48:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=e931bfaf-e2fc-453e-828f-0d0c0831b49f --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.163056/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:48:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:48:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:48:23.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:48:23.725 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:23.724+0000 7f1747f207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:23.727 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:23.724+0000 7f1747f207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:23.728 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:23.724+0000 7f1747f207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:24.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:48:24.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:48:24.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:48:24.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:48:24.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:48:24.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:24.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:48:24.117 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:48:24.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:24.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:24.290 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:24.288+0000 7f1747f207c0 -1 Falling back to public interface 2024-04-25T07:48:24.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:25.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:25.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:25.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:48:25.517 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:48:25.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:25.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:25.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:26.957 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:48:26.956+0000 7f1747f207c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:48:26.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:26.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:26.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:48:26.978 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:48:26.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:26.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:27.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:28.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:28.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:28.683 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:48:28.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:48:28.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:28.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:29.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:30.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:30.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:30.148 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:48:30.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:48:30.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:30.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:30.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:31.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:31.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:31.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:48:31.629 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:48:31.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:31.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:32.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:33.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:33.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:33.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:48:33.099 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:48:33.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:33.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:33.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:34.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:34.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:34.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:48:34.513 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:48:34.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:34.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:35.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:36.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:36.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:36.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:48:36.032 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:48:36.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:36.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:36.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:37.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:37.482 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:48:37.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:48:37.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:37.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:38.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:39.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:39.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:39.043 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:48:39.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:48:39.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:39.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:39.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:40.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:40.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:40.522 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:48:40.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:48:40.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:40.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:40.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:41.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:41.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:41.985 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:48:41.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:48:41.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:41.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:42.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:43.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:43.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:43.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:48:43.842 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:48:43.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:43.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:44.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:45.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:45.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:45.689 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:48:45.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:48:45.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:45.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:46.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:47.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:47.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:47.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:48:47.621 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:48:47.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:47.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:48.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:49.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:49.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:49.767 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:48:49.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:48:49.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:49.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:50.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:51.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:51.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:51.331 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:48:51.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:48:51.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:51.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:52.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:53.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:53.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:53.244 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:48:53.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:48:53.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:53.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:48:53.648 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2629786404,v1:127.0.0.1:6819/2629786404] [v2:127.0.0.1:6820/2629786404,v1:127.0.0.1:6821/2629786404] exists,up 4b8663be-3b1f-4455-b65a-ce4b7e67661b 2024-04-25T07:48:53.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:48:53.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:48:53.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:48:53.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2024-04-25T07:48:53.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=167731 2024-04-25T07:48:53.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2024-04-25T07:48:53.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:48:54.070 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-04-25T07:48:54.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:48:54.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:48:54.643 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' already exists 2024-04-25T07:48:54.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:48:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:48:58.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2024-04-25T07:48:58.147 INFO:tasks.workunit.client.0.smithi102.stdout:hints = 1 2024-04-25T07:48:58.147 INFO:tasks.workunit.client.0.smithi102.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2024-04-25T07:48:58.148 INFO:tasks.workunit.client.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_168726 2024-04-25T07:48:58.148 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:48:58.148 INFO:tasks.workunit.client.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:48:59.148 INFO:tasks.workunit.client.0.smithi102.stdout: 1 16 69 53 0.207005 0.207031 0.277345 0.251588 2024-04-25T07:49:00.148 INFO:tasks.workunit.client.0.smithi102.stdout: 2 16 128 112 0.218718 0.230469 0.267906 0.26652 2024-04-25T07:49:01.148 INFO:tasks.workunit.client.0.smithi102.stdout:Total time run: 2.18177 2024-04-25T07:49:01.148 INFO:tasks.workunit.client.0.smithi102.stdout:Total writes made: 128 2024-04-25T07:49:01.148 INFO:tasks.workunit.client.0.smithi102.stdout:Write size: 4096 2024-04-25T07:49:01.148 INFO:tasks.workunit.client.0.smithi102.stdout:Object size: 4096 2024-04-25T07:49:01.148 INFO:tasks.workunit.client.0.smithi102.stdout:Bandwidth (MB/sec): 0.229172 2024-04-25T07:49:01.148 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Bandwidth: 0.0165728 2024-04-25T07:49:01.148 INFO:tasks.workunit.client.0.smithi102.stdout:Max bandwidth (MB/sec): 0.230469 2024-04-25T07:49:01.149 INFO:tasks.workunit.client.0.smithi102.stdout:Min bandwidth (MB/sec): 0.207031 2024-04-25T07:49:01.149 INFO:tasks.workunit.client.0.smithi102.stdout:Average IOPS: 58 2024-04-25T07:49:01.149 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev IOPS: 4.24264 2024-04-25T07:49:01.149 INFO:tasks.workunit.client.0.smithi102.stdout:Max IOPS: 59 2024-04-25T07:49:01.149 INFO:tasks.workunit.client.0.smithi102.stdout:Min IOPS: 53 2024-04-25T07:49:01.149 INFO:tasks.workunit.client.0.smithi102.stdout:Average Latency(s): 0.266911 2024-04-25T07:49:01.149 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Latency(s): 0.0301111 2024-04-25T07:49:01.149 INFO:tasks.workunit.client.0.smithi102.stdout:Max latency(s): 0.310023 2024-04-25T07:49:01.149 INFO:tasks.workunit.client.0.smithi102.stdout:Min latency(s): 0.136499 2024-04-25T07:49:01.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2024-04-25T07:49:02.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2024-04-25T07:49:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2024-04-25T07:49:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-04-25T07:49:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2024-04-25T07:49:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=164202 2024-04-25T07:49:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 164202 -eq 0 ']' 2024-04-25T07:49:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=164202 2024-04-25T07:49:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 164202 2024-04-25T07:49:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-04-25T07:49:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-04-25T07:49:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-04-25T07:49:02.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-04-25T07:49:03.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-04-25T07:49:03.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-04-25T07:49:03.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-04-25T07:49:03.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-04-25T07:49:03.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-04-25T07:49:03.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 164202 -eq 164202 ']' 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=164202 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 164202 -eq 164202 ']' 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=164202 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 164202 -eq 164202 ']' 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=164202 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 164202 -eq 164202 ']' 2024-04-25T07:49:03.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=167731 2024-04-25T07:49:03.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 167731 -eq 164202 ']' 2024-04-25T07:49:03.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=167731 2024-04-25T07:49:03.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 167731 2024-04-25T07:49:03.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-04-25T07:49:03.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-04-25T07:49:03.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-04-25T07:49:03.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-04-25T07:49:04.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-04-25T07:49:04.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-04-25T07:49:04.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-04-25T07:49:04.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-04-25T07:49:04.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-04-25T07:49:05.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2024-04-25T07:49:05.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2024-04-25T07:49:05.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2024-04-25T07:49:05.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-04-25T07:49:05.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2024-04-25T07:49:05.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2024-04-25T07:49:05.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2024-04-25T07:49:05.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:49:05.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:49:05.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:49:05.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:49:05.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:49:05.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:49:05.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:49:05.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:49:05.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:49:05.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:49:05.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:49:05.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:49:05.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:49:05.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:49:05.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:49:05.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:49:05.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:49:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:49:05.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:49:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:49:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:49:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:49:05.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:49:05.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:05.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:49:05.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.163056 2024-04-25T07:49:05.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-fast-mark-down 0 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:49:05.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:49:05.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:49:05.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:49:05.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:49:05.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:49:05.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:49:05.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:49:05.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:49:05.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:49:05.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:49:05.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:49:05.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:49:05.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:49:05.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:49:05.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:49:05.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:49:05.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:05.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.163056 2024-04-25T07:49:05.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.163056 2024-04-25T07:49:05.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:49:05.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:49:05.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:49:05.407 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:49:05.407 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:49:05.458 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2024-04-25T07:49:05.458 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2024-04-25T07:49:05.510 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:49:05.515 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:49:05.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-force-create-pg 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2024-04-25T07:49:05.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2024-04-25T07:49:05.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none ' 2024-04-25T07:49:05.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2024-04-25T07:49:05.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2024-04-25T07:49:05.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:49:05.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2024-04-25T07:49:05.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2024-04-25T07:49:05.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: setup td/osd-force-create-pg 2024-04-25T07:49:05.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2024-04-25T07:49:05.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2024-04-25T07:49:05.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:49:05.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:49:05.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:49:05.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:49:05.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:49:05.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:49:05.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:49:05.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:49:05.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:49:05.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:49:05.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:49:05.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:49:05.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:49:05.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:49:05.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:49:05.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:49:05.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:49:05.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:49:05.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:49:05.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:49:05.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:49:05.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:49:05.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:49:05.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:05.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:49:05.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168857 2024-04-25T07:49:05.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:49:05.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:49:05.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2024-04-25T07:49:05.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:49:05.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:05.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:49:05.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.168857 2024-04-25T07:49:05.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:49:05.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:49:05.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:49:05.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:49:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2024-04-25T07:49:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:16: run: TEST_reuse_id td/osd-force-create-pg 2024-04-25T07:49:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: local dir=td/osd-force-create-pg 2024-04-25T07:49:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:49:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2024-04-25T07:49:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:49:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:49:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:49:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2024-04-25T07:49:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:49:06.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:49:06.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:49:06.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:49:06.039 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:49:06.039 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:06.039 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:49:06.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:49:06.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:49:06.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:49:06.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:49:06.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:49:06.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:49:06.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:49:06.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:49:06.252 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:49:06.252 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:49:06.252 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:49:06.252 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:49:06.252 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:06.252 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:49:06.252 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.168857/ceph-mon.a.asok 2024-04-25T07:49:06.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:49:06.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.168857/ceph-mon.a.asok config get fsid 2024-04-25T07:49:06.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:49:06.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:49:06.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:49:06.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:49:06.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:49:06.364 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:49:06.364 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:49:06.364 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:49:06.364 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:49:06.364 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:06.364 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:49:06.364 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.168857/ceph-mon.a.asok 2024-04-25T07:49:06.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:49:06.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.168857/ceph-mon.a.asok config get mon_host 2024-04-25T07:49:06.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2024-04-25T07:49:06.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2024-04-25T07:49:06.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:49:06.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:49:06.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:49:06.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2024-04-25T07:49:06.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:49:07.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:49:07.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:49:07.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:49:07.361 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:49:07.361 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:07.361 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:49:07.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:49:07.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:49:07.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:49:07.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:49:07.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:49:07.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:49:07.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:49:07.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:49:07.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:49:07.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:49:07.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:49:07.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/0 2024-04-25T07:49:07.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:49:07.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c5d8e841-cd83-4c85-b57e-223cccaced33 2024-04-25T07:49:07.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c5d8e841-cd83-4c85-b57e-223cccaced33' 2024-04-25T07:49:07.436 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 c5d8e841-cd83-4c85-b57e-223cccaced33 2024-04-25T07:49:07.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:49:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzCipmKAEQGxAALaDNKDHOXyx7CuT6o99KHA== 2024-04-25T07:49:07.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzCipmKAEQGxAALaDNKDHOXyx7CuT6o99KHA=="}' 2024-04-25T07:49:07.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c5d8e841-cd83-4c85-b57e-223cccaced33 -i td/osd-force-create-pg/0/new.json 2024-04-25T07:49:07.786 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:49:07.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/0/new.json 2024-04-25T07:49:07.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzCipmKAEQGxAALaDNKDHOXyx7CuT6o99KHA== --osd-uuid c5d8e841-cd83-4c85-b57e-223cccaced33 2024-04-25T07:49:07.841 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:07.843+0000 7f43006897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:07.863 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:07.863+0000 7f43006897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:07.865 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:07.867+0000 7f43006897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:07.865 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:07.867+0000 7f43006897c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:49:07.865 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:07.867+0000 7f43006897c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:49:07.865 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:07.867+0000 7f43006897c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:49:07.903 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:07.903+0000 7f43006897c0 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2024-04-25T07:49:14.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2024-04-25T07:49:14.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:49:14.270 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:49:14.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:49:14.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:49:14.837 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:49:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:49:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:49:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:49:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:49:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:49:14.867 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:14.867+0000 7f1b48dc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:14.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:14.871+0000 7f1b48dc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:14.870 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:14.871+0000 7f1b48dc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:15.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:49:15.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:49:15.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:49:15.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:49:15.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:49:15.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:15.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:49:15.280 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:49:15.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:15.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:15.430 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:15.431+0000 7f1b48dc27c0 -1 Falling back to public interface 2024-04-25T07:49:15.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:16.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:16.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:16.701 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:49:16.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:49:16.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:16.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:17.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:17.886 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:17.887+0000 7f1b48dc27c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:49:18.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:18.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:18.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:49:18.106 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:49:18.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:18.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:18.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:19.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:19.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:19.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:49:19.748 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:49:19.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:19.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:20.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:21.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:21.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:21.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:49:21.195 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:49:21.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:21.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:21.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:22.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:22.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:22.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:49:22.854 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:49:22.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:22.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:23.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:24.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:24.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:24.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:49:24.263 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:49:24.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:24.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:24.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:25.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:25.748 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:49:25.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:25.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:49:25.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:25.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:26.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:27.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:27.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:27.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:49:27.183 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:49:27.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:27.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:27.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:28.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:28.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:28.766 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:49:28.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:49:28.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:28.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:29.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:30.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:30.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:30.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:49:30.240 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:49:30.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:30.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:30.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:31.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:31.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:31.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:49:31.715 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:49:31.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:31.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:32.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:33.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:33.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:33.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:49:33.945 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:49:33.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:33.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:34.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:35.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:35.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:35.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:49:35.512 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:49:35.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:35.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:36.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:37.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:37.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:37.319 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:49:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:49:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:38.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:39.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:39.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:39.205 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:49:39.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:49:39.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:39.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:40.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:41.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:41.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:41.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:49:41.239 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:49:41.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:41.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:42.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:43.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:43.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:43.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:49:43.217 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:49:43.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:43.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:49:43.624 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/857159435,v1:127.0.0.1:6803/857159435] [v2:127.0.0.1:6804/857159435,v1:127.0.0.1:6805/857159435] exists,up c5d8e841-cd83-4c85-b57e-223cccaced33 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-04-25T07:49:43.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:49:43.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:49:43.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:49:43.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:49:43.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:49:43.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:49:43.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:49:43.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:49:43.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:49:43.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:49:43.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/1 2024-04-25T07:49:43.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:49:43.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef20b331-570f-4cb0-a44b-2223d8b658f1 2024-04-25T07:49:43.631 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 ef20b331-570f-4cb0-a44b-2223d8b658f1 2024-04-25T07:49:43.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ef20b331-570f-4cb0-a44b-2223d8b658f1' 2024-04-25T07:49:43.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:49:43.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAXCypmfnfEJhAARz1bM/qCtZQL3ND6Q3KvAQ== 2024-04-25T07:49:43.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAXCypmfnfEJhAARz1bM/qCtZQL3ND6Q3KvAQ=="}' 2024-04-25T07:49:43.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef20b331-570f-4cb0-a44b-2223d8b658f1 -i td/osd-force-create-pg/1/new.json 2024-04-25T07:49:44.267 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:49:44.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/1/new.json 2024-04-25T07:49:44.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAXCypmfnfEJhAARz1bM/qCtZQL3ND6Q3KvAQ== --osd-uuid ef20b331-570f-4cb0-a44b-2223d8b658f1 2024-04-25T07:49:44.320 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:44.318+0000 7fde28f4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:44.332 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:44.330+0000 7fde28f4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:44.333 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:44.334+0000 7fde28f4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:44.334 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:44.334+0000 7fde28f4f7c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:49:44.334 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:44.334+0000 7fde28f4f7c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:49:44.334 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:44.334+0000 7fde28f4f7c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:49:44.409 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:44.410+0000 7fde28f4f7c0 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2024-04-25T07:49:51.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2024-04-25T07:49:51.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:49:51.248 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:49:51.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:49:51.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:49:51.799 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:49:51.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:49:51.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:49:51.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:49:51.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:49:51.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:49:51.829 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:51.826+0000 7fb4cd7497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:51.856 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:51.854+0000 7fb4cd7497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:51.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:51.858+0000 7fb4cd7497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:52.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:49:52.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:49:52.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:49:52.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:49:52.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:49:52.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:52.258 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:49:52.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:49:52.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:52.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:52.415 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:52.414+0000 7fb4cd7497c0 -1 Falling back to public interface 2024-04-25T07:49:52.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:53.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:53.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:53.656 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:49:53.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:49:53.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:53.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:54.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:55.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:55.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:55.077 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:49:55.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:49:55.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:55.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:55.182 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:49:55.182+0000 7fb4cd7497c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:49:55.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:56.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:56.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:56.542 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:49:56.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:49:56.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:56.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:56.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:57.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:57.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:57.961 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:49:57.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:49:57.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:57.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:58.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:59.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:59.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:59.479 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:49:59.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:49:59.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:59.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:59.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:00.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:00.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:00.946 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:50:00.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:50:00.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:00.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:01.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:02.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:02.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:02.455 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:50:02.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:50:02.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:02.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:02.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:03.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:03.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:03.931 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:50:03.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:50:03.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:03.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:04.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:05.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:05.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:05.468 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:50:05.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:50:05.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:05.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:05.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:06.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:06.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:06.903 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:50:06.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:50:06.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:06.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:07.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:08.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:08.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:08.486 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:50:08.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:50:08.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:08.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:08.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:09.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:09.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:09.974 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:50:09.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:50:09.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:09.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:10.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:11.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:11.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:11.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:50:11.883 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:50:11.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:11.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:12.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:13.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:13.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:13.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:50:13.494 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:50:13.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:13.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:14.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:15.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:15.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:15.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:50:15.307 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:50:15.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:15.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:16.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:17.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:17.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:17.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:50:17.177 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:50:17.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:17.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:18.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:19.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:19.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:19.031 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:50:19.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:50:19.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:19.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:19.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:20.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:20.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:20.833 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:50:20.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:50:20.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:20.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:21.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:22.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:22.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:22.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:50:22.806 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T07:50:22.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:22.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:50:23.216 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/908856382,v1:127.0.0.1:6811/908856382] [v2:127.0.0.1:6812/908856382,v1:127.0.0.1:6813/908856382] exists,up ef20b331-570f-4cb0-a44b-2223d8b658f1 2024-04-25T07:50:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:50:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:50:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:50:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:28: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2024-04-25T07:50:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:50:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:50:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:50:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:50:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2024-04-25T07:50:23.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:50:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:50:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:50:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:50:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-04-25T07:50:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-04-25T07:50:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:50:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:50:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:50:23.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:50:23.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:50:23.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:50:23.221 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:50:23.221 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:23.221 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:50:23.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/2 2024-04-25T07:50:23.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:50:23.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f3b5050a-c230-4041-af84-b79c6442f994 2024-04-25T07:50:23.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f3b5050a-c230-4041-af84-b79c6442f994' 2024-04-25T07:50:23.223 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 f3b5050a-c230-4041-af84-b79c6442f994 2024-04-25T07:50:23.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:50:23.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/Cypm4pGJDhAAziHEWF/0k7utcRioBET+Bg== 2024-04-25T07:50:23.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/Cypm4pGJDhAAziHEWF/0k7utcRioBET+Bg=="}' 2024-04-25T07:50:23.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f3b5050a-c230-4041-af84-b79c6442f994 -i td/osd-force-create-pg/2/new.json 2024-04-25T07:50:23.730 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:50:23.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/2/new.json 2024-04-25T07:50:23.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/Cypm4pGJDhAAziHEWF/0k7utcRioBET+Bg== --osd-uuid f3b5050a-c230-4041-af84-b79c6442f994 2024-04-25T07:50:23.777 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:23.777+0000 7f0489f5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:23.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:23.789+0000 7f0489f5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:23.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:23.793+0000 7f0489f5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:23.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:23.793+0000 7f0489f5c7c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:50:23.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:23.793+0000 7f0489f5c7c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:50:23.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:23.793+0000 7f0489f5c7c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:50:23.885 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:23.885+0000 7f0489f5c7c0 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2024-04-25T07:50:30.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2024-04-25T07:50:30.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:50:30.889 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:50:30.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:50:30.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:50:31.447 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:50:31.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:50:31.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:50:31.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:50:31.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:50:31.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:50:31.477 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:31.473+0000 7efcbfac47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:31.479 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:31.477+0000 7efcbfac47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:31.480 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:31.477+0000 7efcbfac47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:31.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:50:31.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:50:31.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:50:31.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:50:31.892 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:50:31.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:50:31.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:31.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:50:31.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:31.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:32.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:32.037+0000 7efcbfac47c0 -1 Falling back to public interface 2024-04-25T07:50:32.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:33.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:33.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:33.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:50:33.352 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:50:33.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:33.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:33.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:34.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:34.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:34.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:50:34.764 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:50:34.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:34.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:34.829 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:50:34.829+0000 7efcbfac47c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:50:35.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:36.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:36.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:36.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:50:36.220 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:50:36.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:36.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:36.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:37.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:37.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:37.809 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:50:37.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:50:37.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:37.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:38.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:39.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:39.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:39.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:50:39.355 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:50:39.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:39.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:39.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:40.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:40.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:40.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:50:40.911 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:50:40.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:40.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:41.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:42.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:42.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:42.362 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:50:42.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:50:42.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:42.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:42.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:43.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:43.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:43.881 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:50:43.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:50:43.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:43.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:44.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:45.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:45.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:45.311 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:50:45.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:50:45.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:45.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:45.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:46.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:46.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:46.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:50:46.781 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:50:46.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:46.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:47.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:48.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:48.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:48.186 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:50:48.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:50:48.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:48.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:48.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:49.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:49.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:49.785 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:50:49.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:50:49.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:49.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:50.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:51.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:51.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:51.467 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:50:51.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:50:51.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:51.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:52.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:53.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:53.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:53.245 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:50:53.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:50:53.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:53.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:53.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:54.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:54.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:54.973 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:50:54.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:50:54.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:54.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:55.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:56.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:56.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:56.929 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:50:56.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:50:56.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:56.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:58.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:58.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:58.670 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:50:58.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:50:58.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:58.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:59.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:00.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:00.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:00.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:51:00.936 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:51:00.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:00.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:51:01.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:02.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:02.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:02.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:51:02.377 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T07:51:02.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:02.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:51:02.797 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4051632850,v1:127.0.0.1:6819/4051632850] [v2:127.0.0.1:6820/4051632850,v1:127.0.0.1:6821/4051632850] exists,up f3b5050a-c230-4041-af84-b79c6442f994 2024-04-25T07:51:02.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:51:02.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:51:02.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:51:02.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: ceph osd pool create foo 50 2024-04-25T07:51:03.332 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T07:51:03.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:31: TEST_reuse_id: wait_for_clean 2024-04-25T07:51:03.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:51:03.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:51:03.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:51:03.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:51:03.350 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:51:03.351 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:51:03.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:51:03.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:51:03.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:51:03.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:51:03.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:51:03.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:51:03.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:51:03.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:51:03.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:51:04.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:51:04.198 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:51:04.199 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:51:04.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:51:04.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:51:04.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:51:04.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836503 2024-04-25T07:51:04.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836503 2024-04-25T07:51:04.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503' 2024-04-25T07:51:04.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:51:04.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:51:04.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672980 2024-04-25T07:51:04.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672980 2024-04-25T07:51:04.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672980' 2024-04-25T07:51:04.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:51:04.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:51:04.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509449 2024-04-25T07:51:04.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509449 2024-04-25T07:51:04.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672980 2-64424509449' 2024-04-25T07:51:04.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:51:04.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836503 2024-04-25T07:51:04.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:51:04.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:51:04.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836503 2024-04-25T07:51:04.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:51:04.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836503 2024-04-25T07:51:04.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-04-25T07:51:04.755 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836503 2024-04-25T07:51:04.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:51:05.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836503 2024-04-25T07:51:05.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:51:06.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:51:06.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:51:06.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836503 2024-04-25T07:51:06.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:51:06.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672980 2024-04-25T07:51:06.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:51:06.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:51:06.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672980 2024-04-25T07:51:06.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:51:06.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672980 2024-04-25T07:51:06.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-04-25T07:51:06.718 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672980 2024-04-25T07:51:06.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:51:07.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672980 -lt 42949672980 2024-04-25T07:51:07.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:51:07.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509449 2024-04-25T07:51:07.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:51:07.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:51:07.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509449 2024-04-25T07:51:07.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:51:07.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509449 2024-04-25T07:51:07.129 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509449 2024-04-25T07:51:07.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-04-25T07:51:07.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:51:07.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509449 2024-04-25T07:51:07.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:51:07.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:51:07.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:51:08.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-04-25T07:51:08.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:51:08.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:51:08.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:51:08.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:51:08.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:51:08.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:51:08.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:51:08.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=36 2024-04-25T07:51:08.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:51:08.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:51:08.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:51:09.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 36 = 50 2024-04-25T07:51:09.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 36 '!=' -1 2024-04-25T07:51:09.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:51:09.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=36 2024-04-25T07:51:09.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:51:09.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:51:09.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:51:09.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:51:09.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:51:09.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:51:09.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:51:09.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:51:09.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:51:09.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:51:09.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=36 2024-04-25T07:51:09.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:51:09.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:51:09.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:51:10.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 36 = 50 2024-04-25T07:51:10.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 36 '!=' 36 2024-04-25T07:51:10.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:51:10.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:51:10.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:51:10.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:51:10.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:51:10.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:51:10.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:51:10.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:51:10.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:51:10.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T07:51:10.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:51:10.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:51:10.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:51:10.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:51:10.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:51:10.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:51:10.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:51:10.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:51:10.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:51:10.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:51:11.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-04-25T07:51:11.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:51:11.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:51:11.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:51:11.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-04-25T07:51:11.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:51:11.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:51:11.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2024-04-25T07:51:11.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:51:11.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:51:11.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:51:11.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:51:11.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:51:12.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:51:12.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2024-04-25T07:51:12.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:51:12.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:51:12.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:51:12.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:51:12.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:51:13.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:51:13.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2024-04-25T07:51:13.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:51:13.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:51:13.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:51:13.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:51:13.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:51:13.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:51:13.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2024-04-25T07:51:15.147 INFO:tasks.workunit.client.0.smithi102.stderr:PG '1.0' not found 2024-04-25T07:51:15.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2024-04-25T07:51:17.135 INFO:tasks.workunit.client.0.smithi102.stdout: marking collection for removal 2024-04-25T07:51:17.135 INFO:tasks.workunit.client.0.smithi102.stdout:setting '_remove' omap key 2024-04-25T07:51:17.135 INFO:tasks.workunit.client.0.smithi102.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T07:51:17.309 INFO:tasks.workunit.client.0.smithi102.stdout:Remove successful 2024-04-25T07:51:17.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2024-04-25T07:51:19.181 INFO:tasks.workunit.client.0.smithi102.stderr:PG '1.0' not found 2024-04-25T07:51:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2024-04-25T07:51:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:51:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:51:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:51:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:51:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/0 2024-04-25T07:51:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:51:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:51:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:51:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:51:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:51:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:51:19.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:51:19.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:51:19.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:51:19.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:51:19.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:51:19.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:51:19.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:51:19.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:51:19.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/0 2024-04-25T07:51:19.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:51:19.476 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:51:19.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:51:19.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/0/whoami 2024-04-25T07:51:19.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:51:19.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:51:19.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:51:19.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:51:19.506 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:19.504+0000 7f1c424cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:19.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:19.508+0000 7f1c424cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:19.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:19.508+0000 7f1c424cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:19.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:51:19.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:51:19.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:51:19.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:51:19.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:51:19.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:19.945 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:51:19.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:51:19.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:19.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:20.062 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:20.060+0000 7f1c424cf7c0 -1 Falling back to public interface 2024-04-25T07:51:20.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:21.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:21.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:21.452 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:51:21.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:51:21.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:21.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:21.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:22.838 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:22.836+0000 7f1c424cf7c0 -1 osd.0 24 log_to_monitors true 2024-04-25T07:51:22.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:22.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:22.868 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:51:22.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:51:22.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:22.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:23.402 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 30 up_thru 18 down_at 25 last_clean_interval [5,24) [v2:127.0.0.1:6802/3286916975,v1:127.0.0.1:6803/3286916975] [v2:127.0.0.1:6804/3286916975,v1:127.0.0.1:6805/3286916975] exists,up c5d8e841-cd83-4c85-b57e-223cccaced33 2024-04-25T07:51:23.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/1 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:51:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:51:23.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:51:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:51:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:51:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:51:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:51:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:51:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:51:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:51:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/1 2024-04-25T07:51:23.408 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:51:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:51:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:51:23.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/1/whoami 2024-04-25T07:51:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:51:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:51:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:51:23.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:51:23.435 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:23.432+0000 7f4edae1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:23.486 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:23.484+0000 7f4edae1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:23.487 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:23.484+0000 7f4edae1c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:23.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:51:23.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:51:23.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:51:23.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:51:23.939 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:51:23.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:51:23.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:23.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:51:23.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:23.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:24.055 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:24.052+0000 7f4edae1c7c0 -1 Falling back to public interface 2024-04-25T07:51:24.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:25.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:25.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:25.379 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:51:25.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:51:25.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:25.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:25.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:26.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:26.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:26.791 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:51:26.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:51:26.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:26.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:26.931 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:26.928+0000 7f4edae1c7c0 -1 osd.1 25 log_to_monitors true 2024-04-25T07:51:27.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:28.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:28.279 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:51:28.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:28.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:51:28.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:28.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:28.858 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 33 up_thru 33 down_at 27 last_clean_interval [10,25) [v2:127.0.0.1:6810/3692821600,v1:127.0.0.1:6811/3692821600] [v2:127.0.0.1:6812/3692821600,v1:127.0.0.1:6813/3692821600] exists,up ef20b331-570f-4cb0-a44b-2223d8b658f1 2024-04-25T07:51:28.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:51:28.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:51:28.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:51:28.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2024-04-25T07:51:28.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:51:28.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/2 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:51:28.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:51:28.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:51:28.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/2 2024-04-25T07:51:28.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:51:28.865 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:51:28.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=36044a9f-ccd7-46a7-808b-17252d1772a1 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168857/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:51:28.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/2/whoami 2024-04-25T07:51:28.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:51:28.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:51:28.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:51:28.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:51:28.892 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:28.888+0000 7f68f69137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:28.920 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:28.916+0000 7f68f69137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:28.921 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:28.920+0000 7f68f69137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:51:29.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:51:29.313 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:51:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:51:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:51:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:51:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:51:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:51:29.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:29.495 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:29.492+0000 7f68f69137c0 -1 Falling back to public interface 2024-04-25T07:51:29.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:30.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:30.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:30.735 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:51:30.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:51:30.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:30.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:51:31.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:32.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:32.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:32.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:51:32.159 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:51:32.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:32.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:51:32.198 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:32.196+0000 7f68f69137c0 -1 osd.2 27 log_to_monitors true 2024-04-25T07:51:32.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:33.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:33.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:33.661 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:51:33.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:51:33.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:33.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:51:34.081 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 36 up_thru 36 down_at 28 last_clean_interval [15,27) [v2:127.0.0.1:6818/3896611926,v1:127.0.0.1:6819/3896611926] [v2:127.0.0.1:6820/3896611926,v1:127.0.0.1:6821/3896611926] exists,up f3b5050a-c230-4041-af84-b79c6442f994 2024-04-25T07:51:34.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:51:34.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:51:34.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:51:34.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: sleep 10 2024-04-25T07:51:44.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: ceph pg ls 2024-04-25T07:51:44.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep 1.0 2024-04-25T07:51:44.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep stale 2024-04-25T07:51:44.489 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 stale+active+clean 39s 0'0 24:18 [1]p1 [1]p1 2024-04-25T07:51:03.153962+0000 2024-04-25T07:51:03.153962+0000 0 periodic scrub scheduled @ 2024-04-26T13:25:42.365278+0000 2024-04-25T07:51:44.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2024-04-25T07:51:45.036 INFO:tasks.workunit.client.0.smithi102.stderr:pg 1.0 already creating 2024-04-25T07:51:45.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:46: TEST_reuse_id: wait_for_clean 2024-04-25T07:51:45.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:51:45.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:51:45.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:51:45.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:51:45.053 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:51:45.053 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:51:45.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:51:45.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:51:45.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:51:45.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:51:45.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:51:45.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:51:45.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:51:45.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:51:45.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:51:45.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:51:45.644 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T07:51:45.645 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T07:51:45.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:51:45.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:51:45.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:51:45.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018887 2024-04-25T07:51:45.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018887 2024-04-25T07:51:45.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018887' 2024-04-25T07:51:45.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:51:45.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:51:45.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=141733920774 2024-04-25T07:51:45.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 141733920774 2024-04-25T07:51:45.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018887 1-141733920774' 2024-04-25T07:51:45.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:51:45.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:51:46.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822661 2024-04-25T07:51:46.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822661 2024-04-25T07:51:46.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-128849018887 1-141733920774 2-154618822661' 2024-04-25T07:51:46.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:51:46.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-128849018887 2024-04-25T07:51:46.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:51:46.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:51:46.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-128849018887 2024-04-25T07:51:46.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:51:46.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018887 2024-04-25T07:51:46.085 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 128849018887 2024-04-25T07:51:46.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 128849018887' 2024-04-25T07:51:46.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:51:46.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018887 -lt 128849018887 2024-04-25T07:51:46.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:51:46.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-141733920774 2024-04-25T07:51:46.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:51:46.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:51:46.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-141733920774 2024-04-25T07:51:46.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:51:46.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=141733920774 2024-04-25T07:51:46.470 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 141733920774 2024-04-25T07:51:46.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 141733920774' 2024-04-25T07:51:46.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:51:46.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 141733920774 -lt 141733920774 2024-04-25T07:51:46.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:51:46.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-154618822661 2024-04-25T07:51:46.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:51:46.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:51:46.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-154618822661 2024-04-25T07:51:46.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:51:46.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822661 2024-04-25T07:51:46.876 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 154618822661 2024-04-25T07:51:46.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 154618822661' 2024-04-25T07:51:46.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:51:47.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822659 -lt 154618822661 2024-04-25T07:51:47.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:51:48.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:51:48.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:51:48.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822662 -lt 154618822661 2024-04-25T07:51:48.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:51:48.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:51:48.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:51:49.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-04-25T07:51:49.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:51:49.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:51:49.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:51:49.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:51:49.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:51:49.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:51:49.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:51:49.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-04-25T07:51:49.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:51:49.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:51:49.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:51:50.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-04-25T07:51:50.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:51:50.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:51:50.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:17: run: teardown td/osd-force-create-pg 2024-04-25T07:51:50.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:51:50.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:51:50.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:51:50.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:51:50.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:51:50.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:51:50.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:51:50.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:51:50.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:51:50.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:51:50.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:51:50.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:51:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:51:50.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:51:50.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:51:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:51:50.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:51:50.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:51:50.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:51:50.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:51:50.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:51:50.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:51:50.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:51:50.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:50.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:51:50.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168857 2024-04-25T07:51:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:51:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:51:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:51:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-force-create-pg 0 2024-04-25T07:51:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:51:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:51:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:51:50.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:51:50.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:51:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:51:50.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:51:50.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:51:50.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:51:50.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:51:50.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:51:50.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:51:50.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:51:50.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:51:50.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:51:50.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:51:50.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:51:50.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:51:50.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:51:50.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:51:50.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:51:50.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:51:50.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:51:50.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:50.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168857 2024-04-25T07:51:50.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168857 2024-04-25T07:51:50.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:51:50.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:51:50.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:51:50.350 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:51:50.350 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:51:50.402 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2024-04-25T07:51:50.402 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2024-04-25T07:51:50.456 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:51:50.462 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:51:50.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:51:50.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:51:50.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-markdown 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none ' 2024-04-25T07:51:50.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2024-04-25T07:51:50.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2024-04-25T07:51:50.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:51:50.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2024-04-25T07:51:50.465 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_markdown_boot_exceed_time 2024-04-25T07:51:50.465 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_markdown_exceed_maxdown_count 2024-04-25T07:51:50.465 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_osd_stop' 2024-04-25T07:51:50.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:51:50.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:51:50.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:51:50.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:51:50.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:51:50.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:51:50.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:51:50.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:51:50.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:51:50.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:51:50.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:51:50.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:51:50.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:51:50.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:51:50.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:51:50.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:51:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:51:50.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:51:50.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:51:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:51:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:51:50.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:51:50.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:51:50.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:51:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:51:50.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:51:50.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:51:50.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:50.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:51:50.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178198 2024-04-25T07:51:50.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:51:50.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:51:50.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:51:50.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:51:50.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:50.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:51:50.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.178198 2024-04-25T07:51:50.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:51:50.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:51:50.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:51:50.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:51:50.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:51:50.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2024-04-25T07:51:50.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2024-04-25T07:51:50.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2024-04-25T07:51:50.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:51:50.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:51:50.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:51:50.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:51:50.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:51:50.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:51:50.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:51:50.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:50.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:50.940 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:50.940 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:50.940 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:51:50.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:51:50.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:51:51.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:51:51.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:51:51.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:51:51.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:51:51.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:51:51.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:51:51.154 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:51:51.154 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:51:51.154 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:51:51.154 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:51:51.154 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:51.154 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:51:51.154 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178198/ceph-mon.a.asok 2024-04-25T07:51:51.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:51:51.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178198/ceph-mon.a.asok config get fsid 2024-04-25T07:51:51.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:51:51.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:51:51.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:51:51.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:51:51.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:51:51.280 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:51:51.280 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:51:51.280 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:51:51.280 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:51:51.280 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:51.280 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:51:51.281 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178198/ceph-mon.a.asok 2024-04-25T07:51:51.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:51:51.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178198/ceph-mon.a.asok config get mon_host 2024-04-25T07:51:51.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2024-04-25T07:51:51.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:51:51.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:51:51.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:51:51.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:51:51.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:51:51.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:51:52.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:51:52.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:52.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:52.304 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:52.304 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:52.304 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:51:52.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:51:52.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:51:52.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:51:52.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2024-04-25T07:51:52.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:51:52.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:51:52.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:51:52.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:51:52.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:51:52.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:51:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:51:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:51:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:51:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:51:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:51:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:51:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:51:52.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:51:52.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:51:52.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:51:52.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:51:52.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:51:52.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:51:52.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:51:52.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:51:52.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:51:52.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:51:52.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:51:52.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=34cec229-009f-4650-aa84-a3ba3ae2c587 2024-04-25T07:51:52.343 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 34cec229-009f-4650-aa84-a3ba3ae2c587 2024-04-25T07:51:52.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 34cec229-009f-4650-aa84-a3ba3ae2c587' 2024-04-25T07:51:52.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:51:52.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCYCypmZZlyFRAAOR1PpQCjhVO6DJ0/l6C5CA== 2024-04-25T07:51:52.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCYCypmZZlyFRAAOR1PpQCjhVO6DJ0/l6C5CA=="}' 2024-04-25T07:51:52.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 34cec229-009f-4650-aa84-a3ba3ae2c587 -i td/osd-markdown/0/new.json 2024-04-25T07:51:52.687 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:51:52.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:51:52.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYCypmZZlyFRAAOR1PpQCjhVO6DJ0/l6C5CA== --osd-uuid 34cec229-009f-4650-aa84-a3ba3ae2c587 2024-04-25T07:51:52.735 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:52.732+0000 7f1b0893b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:52.764 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:52.760+0000 7f1b0893b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:52.765 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:52.764+0000 7f1b0893b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:52.766 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:52.764+0000 7f1b0893b7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:51:52.766 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:52.764+0000 7f1b0893b7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:51:52.766 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:52.764+0000 7f1b0893b7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:51:52.804 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:52.800+0000 7f1b0893b7c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:51:59.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:51:59.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:51:59.292 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:51:59.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:51:59.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:51:59.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:51:59.843 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:51:59.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:51:59.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:51:59.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:51:59.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:51:59.873 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:59.871+0000 7f970648a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:59.875 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:59.875+0000 7f970648a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:59.876 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:51:59.875+0000 7f970648a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:00.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:52:00.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:52:00.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:52:00.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:52:00.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:52:00.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:00.250 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:52:00.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:52:00.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:00.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:00.431 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:00.431+0000 7f970648a7c0 -1 Falling back to public interface 2024-04-25T07:52:00.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:01.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:01.694 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:52:01.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:01.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:52:01.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:01.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:02.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:03.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:03.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:03.098 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:52:03.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:52:03.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:03.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:03.115 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:03.115+0000 7f970648a7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:52:03.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:04.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:04.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:04.609 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:52:04.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:52:04.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:04.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:05.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:06.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:06.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:06.067 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:52:06.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:52:06.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:06.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:06.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:07.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:07.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:07.491 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:52:07.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:52:07.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:07.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:07.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:08.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:08.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:08.995 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:52:08.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:52:08.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:08.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:09.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:10.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:10.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:10.417 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:52:10.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:52:10.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:10.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:10.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:11.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:11.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:11.825 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:52:11.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:52:11.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:11.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:12.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:13.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:13.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:13.266 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:52:13.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:52:13.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:13.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:14.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:14.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:14.693 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:52:14.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:52:14.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:14.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:15.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:16.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:16.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:16.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:52:16.301 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:52:16.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:16.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:16.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:17.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:17.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:17.949 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:52:17.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:52:17.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:17.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:18.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:19.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:19.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:19.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:52:19.884 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:52:19.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:19.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:20.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:21.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:21.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:21.651 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:52:21.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:52:21.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:21.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:22.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:23.477 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:52:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:52:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:23.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:24.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:25.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:25.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:25.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:52:25.378 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:52:25.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:25.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:26.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:27.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:27.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:27.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:52:27.195 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:52:27.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:27.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:27.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:28.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:28.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:28.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:52:28.994 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:52:28.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:28.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:52:29.408 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3996603757,v1:127.0.0.1:6803/3996603757] [v2:127.0.0.1:6804/3996603757,v1:127.0.0.1:6805/3996603757] exists,up 34cec229-009f-4650-aa84-a3ba3ae2c587 2024-04-25T07:52:29.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:52:29.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:52:29.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:52:29.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:52:29.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:52:29.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:52:29.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:52:29.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:52:29.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:52:29.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:52:29.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:52:29.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f88485fa-5217-4795-b5f4-798707760d5d 2024-04-25T07:52:29.414 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 f88485fa-5217-4795-b5f4-798707760d5d 2024-04-25T07:52:29.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f88485fa-5217-4795-b5f4-798707760d5d' 2024-04-25T07:52:29.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:52:29.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9CypmOYHxGRAAEvBOFwJT3+rtpkIynVo8cg== 2024-04-25T07:52:29.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9CypmOYHxGRAAEvBOFwJT3+rtpkIynVo8cg=="}' 2024-04-25T07:52:29.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f88485fa-5217-4795-b5f4-798707760d5d -i td/osd-markdown/1/new.json 2024-04-25T07:52:29.897 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:52:29.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:52:29.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9CypmOYHxGRAAEvBOFwJT3+rtpkIynVo8cg== --osd-uuid f88485fa-5217-4795-b5f4-798707760d5d 2024-04-25T07:52:29.938 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:29.939+0000 7fa8cb8187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:29.976 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:29.975+0000 7fa8cb8187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:29.977 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:29.979+0000 7fa8cb8187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:29.978 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:29.979+0000 7fa8cb8187c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:52:29.978 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:29.979+0000 7fa8cb8187c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:52:29.978 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:29.979+0000 7fa8cb8187c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:52:30.007 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:30.007+0000 7fa8cb8187c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:52:36.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:52:36.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:52:36.788 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:52:36.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:52:36.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:52:37.339 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:52:37.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:52:37.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:52:37.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:52:37.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:52:37.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:52:37.369 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:37.367+0000 7fe142a487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:37.371 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:37.371+0000 7fe142a487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:37.372 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:37.371+0000 7fe142a487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:37.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:52:37.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:52:37.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:52:37.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:52:37.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:52:37.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:37.768 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:52:37.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:52:37.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:37.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:37.966 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:37.967+0000 7fe142a487c0 -1 Falling back to public interface 2024-04-25T07:52:38.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:39.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:39.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:39.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:52:39.198 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:52:39.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:39.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:39.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:40.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:40.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:40.627 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:52:40.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:52:40.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:40.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:40.671 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:52:40.671+0000 7fe142a487c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:52:41.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:42.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:42.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:42.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:52:42.291 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:52:42.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:42.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:42.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:43.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:43.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:43.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:52:43.814 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:52:43.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:43.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:44.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:45.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:45.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:45.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:52:45.287 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:52:45.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:45.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:45.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:46.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:46.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:46.775 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:52:46.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:52:46.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:46.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:47.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:48.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:48.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:48.301 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:52:48.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:52:48.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:48.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:48.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:49.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:49.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:49.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:52:49.795 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:52:49.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:49.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:50.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:51.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:51.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:51.310 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:52:51.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:52:51.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:51.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:51.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:52.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:52.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:52.723 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:52:52.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:52:52.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:52.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:53.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:54.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:54.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:54.202 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:52:54.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:52:54.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:54.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:54.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:55.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:55.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:55.707 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:52:55.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:52:55.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:55.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:56.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:57.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:57.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:57.825 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:52:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:52:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:58.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:59.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:59.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:59.747 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:52:59.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:52:59.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:59.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:00.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:01.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:01.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:01.614 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:53:01.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:53:01.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:01.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:02.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:03.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:03.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:03.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:53:03.604 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:53:03.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:03.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:04.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:05.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:05.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:05.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:53:05.137 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:53:05.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:05.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:06.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:07.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:07.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:07.006 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:53:07.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:53:07.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:07.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:07.446 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/983334241,v1:127.0.0.1:6811/983334241] [v2:127.0.0.1:6812/983334241,v1:127.0.0.1:6813/983334241] exists,up f88485fa-5217-4795-b5f4-798707760d5d 2024-04-25T07:53:07.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:53:07.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:53:07.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:53:07.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2024-04-25T07:53:07.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:53:07.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:53:07.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:53:07.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:53:07.451 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:53:07.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:53:07.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:53:07.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:53:07.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9370f45f-70d2-439e-8cff-5cdc7bac9294 2024-04-25T07:53:07.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9370f45f-70d2-439e-8cff-5cdc7bac9294' 2024-04-25T07:53:07.454 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 9370f45f-70d2-439e-8cff-5cdc7bac9294 2024-04-25T07:53:07.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:53:07.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDjCypmCrFEHBAA7rx9MJowyX3qE3AkBBb4bg== 2024-04-25T07:53:07.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDjCypmCrFEHBAA7rx9MJowyX3qE3AkBBb4bg=="}' 2024-04-25T07:53:07.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9370f45f-70d2-439e-8cff-5cdc7bac9294 -i td/osd-markdown/2/new.json 2024-04-25T07:53:07.963 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:53:07.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:53:07.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjCypmCrFEHBAA7rx9MJowyX3qE3AkBBb4bg== --osd-uuid 9370f45f-70d2-439e-8cff-5cdc7bac9294 2024-04-25T07:53:08.011 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:08.010+0000 7f8706ca57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:08.051 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:08.050+0000 7f8706ca57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:08.052 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:08.050+0000 7f8706ca57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:08.053 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:08.050+0000 7f8706ca57c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:53:08.053 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:08.050+0000 7f8706ca57c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:53:08.053 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:08.050+0000 7f8706ca57c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:53:08.081 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:08.082+0000 7f8706ca57c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:53:14.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:53:14.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:53:14.907 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:53:14.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:53:14.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:53:15.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:53:15.459 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:53:15.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:53:15.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:53:15.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:53:15.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:53:15.487 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:15.486+0000 7f5aa26f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:15.490 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:15.490+0000 7f5aa26f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:15.491 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:15.490+0000 7f5aa26f77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:15.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:53:15.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:53:15.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:53:15.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:53:15.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:53:15.839 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:53:15.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:15.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:53:15.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:15.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:16.064 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:16.062+0000 7f5aa26f77c0 -1 Falling back to public interface 2024-04-25T07:53:16.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:17.269 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:53:17.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:17.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:17.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:53:17.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:17.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:17.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:18.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:18.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:18.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:53:18.675 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:53:18.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:18.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:18.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:53:18.686+0000 7f5aa26f77c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:53:19.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:20.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:20.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:20.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:53:20.126 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:53:20.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:20.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:20.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:21.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:21.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:21.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:53:21.564 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:53:21.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:21.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:22.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:23.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:23.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:23.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:53:23.163 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:53:23.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:23.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:23.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:24.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:24.706 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:53:24.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:53:24.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:24.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:25.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:26.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:26.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:26.236 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:53:26.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:53:26.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:26.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:26.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:27.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:27.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:27.645 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:53:27.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:53:27.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:27.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:28.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:29.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:29.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:29.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:53:29.166 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:53:29.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:29.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:29.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:30.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:30.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:30.697 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:53:30.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:53:30.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:30.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:31.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:32.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:32.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:32.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:53:32.152 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:53:32.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:32.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:32.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:33.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:33.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:33.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:53:33.612 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:53:33.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:33.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:34.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:35.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:35.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:35.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:53:35.518 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:53:35.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:35.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:36.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:37.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:37.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:37.858 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:53:37.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:53:37.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:37.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:38.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:39.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:39.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:39.685 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:53:39.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:53:39.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:39.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:40.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:41.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:41.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:41.407 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:53:41.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:53:41.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:41.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:42.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:43.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:43.019 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:53:43.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:43.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:53:43.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:43.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:43.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:44.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:44.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:44.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:53:44.505 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:53:44.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:44.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:45.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:46.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:46.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:46.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:53:46.087 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T07:53:46.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:46.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:46.493 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2095892625,v1:127.0.0.1:6819/2095892625] [v2:127.0.0.1:6820/2095892625,v1:127.0.0.1:6821/2095892625] exists,up 9370f45f-70d2-439e-8cff-5cdc7bac9294 2024-04-25T07:53:46.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:53:46.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:53:46.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:53:46.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2024-04-25T07:53:46.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:53:46.910 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-04-25T07:53:46.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:53:46.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:53:47.489 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' already exists 2024-04-25T07:53:47.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:53:48.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:53:51.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2024-04-25T07:53:51.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2024-04-25T07:53:51.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2024-04-25T07:53:51.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T07:53:51.579 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T07:53:51.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2024-04-25T07:53:51.733 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T07:53:51.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2024-04-25T07:53:51.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2024-04-25T07:53:51.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2024-04-25T07:53:51.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T07:53:51.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2024-04-25T07:53:51.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:53:51.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:53:51.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:53:52.295 INFO:tasks.workunit.client.0.smithi102.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:53:52.295 INFO:tasks.workunit.client.0.smithi102.stdout:-1 0.29306 root default 2024-04-25T07:53:52.296 INFO:tasks.workunit.client.0.smithi102.stdout:-3 0.29306 host smithi102 2024-04-25T07:53:52.296 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:53:52.296 INFO:tasks.workunit.client.0.smithi102.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:53:52.296 INFO:tasks.workunit.client.0.smithi102.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:53:52.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:53:52.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:53:52.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:53:52.738 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:53:52.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:53:52.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:53:53.241 INFO:tasks.workunit.client.0.smithi102.stderr:marked down osd.0. 2024-04-25T07:53:53.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:54:03.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:54:03.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:54:03.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:54:03.853 INFO:tasks.workunit.client.0.smithi102.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:54:03.853 INFO:tasks.workunit.client.0.smithi102.stdout:-1 0.29306 root default 2024-04-25T07:54:03.853 INFO:tasks.workunit.client.0.smithi102.stdout:-3 0.29306 host smithi102 2024-04-25T07:54:03.853 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:03.853 INFO:tasks.workunit.client.0.smithi102.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:54:03.853 INFO:tasks.workunit.client.0.smithi102.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:54:03.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:54:03.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:54:03.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:54:04.310 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:04.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:54:04.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:54:04.763 INFO:tasks.workunit.client.0.smithi102.stderr:marked down osd.0. 2024-04-25T07:54:04.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:54:14.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:54:14.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:54:14.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:54:15.350 INFO:tasks.workunit.client.0.smithi102.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:54:15.351 INFO:tasks.workunit.client.0.smithi102.stdout:-1 0.29306 root default 2024-04-25T07:54:15.351 INFO:tasks.workunit.client.0.smithi102.stdout:-3 0.29306 host smithi102 2024-04-25T07:54:15.351 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:15.351 INFO:tasks.workunit.client.0.smithi102.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:54:15.351 INFO:tasks.workunit.client.0.smithi102.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:54:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:54:15.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:54:15.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:54:15.803 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:15.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:54:15.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:54:16.278 INFO:tasks.workunit.client.0.smithi102.stderr:marked down osd.0. 2024-04-25T07:54:16.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:54:26.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2024-04-25T07:54:41.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:54:41.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2024-04-25T07:54:41.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2024-04-25T07:54:41.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2024-04-25T07:54:41.917 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:54:41.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T07:54:41.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:54:41.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:54:41.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:54:41.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:54:41.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:54:41.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:54:41.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:54:41.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:54:42.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:54:42.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:54:42.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:54:42.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:54:42.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:54:42.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:54:42.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:54:42.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:54:42.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:54:42.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:54:42.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:54:42.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:54:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:54:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:54:42.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:54:42.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:42.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:54:42.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178198 2024-04-25T07:54:42.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:54:42.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:54:42.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:54:42.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:54:42.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:54:42.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:54:42.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:54:42.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:54:42.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:54:42.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:54:42.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:54:42.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:54:42.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:54:42.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:54:42.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:54:42.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:54:42.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:54:42.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:54:42.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:54:42.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:54:42.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:54:42.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:54:42.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:54:42.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:54:42.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:54:42.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:54:42.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:54:42.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:54:42.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:54:42.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:42.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:54:42.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178198 2024-04-25T07:54:42.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:54:42.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:54:42.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:54:42.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:54:42.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:42.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:54:42.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.178198 2024-04-25T07:54:42.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:54:42.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:54:42.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:54:42.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:54:42.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:54:42.766 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:54:42.767 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:42.767 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:54:42.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:54:42.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:54:42.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:54:42.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:54:42.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:54:42.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:54:42.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:54:42.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:54:42.988 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:54:42.988 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:54:42.988 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:54:42.989 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:54:42.989 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:42.989 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:54:42.989 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178198/ceph-mon.a.asok 2024-04-25T07:54:42.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:54:42.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178198/ceph-mon.a.asok config get fsid 2024-04-25T07:54:43.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:54:43.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:54:43.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:54:43.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:54:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:54:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:54:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:54:43.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:54:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:54:43.108 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:43.109 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:54:43.109 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178198/ceph-mon.a.asok 2024-04-25T07:54:43.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:54:43.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178198/ceph-mon.a.asok config get mon_host 2024-04-25T07:54:43.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2024-04-25T07:54:43.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:54:43.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:54:43.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:54:43.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:54:43.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:54:43.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:54:44.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:54:44.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:54:44.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:54:44.133 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:54:44.133 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:44.133 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:54:44.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:54:44.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:54:44.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:54:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:54:44.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:54:44.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:54:44.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:54:44.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:54:44.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:54:44.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:54:44.170 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:54:44.170 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:44.170 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:54:44.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:54:44.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:54:44.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c6820445-f1b7-4719-b27f-022a76df9a3b 2024-04-25T07:54:44.173 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 c6820445-f1b7-4719-b27f-022a76df9a3b 2024-04-25T07:54:44.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c6820445-f1b7-4719-b27f-022a76df9a3b' 2024-04-25T07:54:44.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:54:44.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEDCpmG6NPCxAA0mclRcB/Uldfvdy2MxRGZA== 2024-04-25T07:54:44.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEDCpmG6NPCxAA0mclRcB/Uldfvdy2MxRGZA=="}' 2024-04-25T07:54:44.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c6820445-f1b7-4719-b27f-022a76df9a3b -i td/osd-markdown/0/new.json 2024-04-25T07:54:44.514 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:54:44.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:54:44.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEDCpmG6NPCxAA0mclRcB/Uldfvdy2MxRGZA== --osd-uuid c6820445-f1b7-4719-b27f-022a76df9a3b 2024-04-25T07:54:44.564 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:44.560+0000 7fa85ee4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:44.583 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:44.580+0000 7fa85ee4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:44.584 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:44.580+0000 7fa85ee4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:44.585 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:44.580+0000 7fa85ee4e7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:54:44.585 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:44.580+0000 7fa85ee4e7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:54:44.585 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:44.580+0000 7fa85ee4e7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:54:44.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:44.620+0000 7fa85ee4e7c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:54:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:54:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:54:50.988 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:54:50.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:54:50.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:54:51.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:54:51.550 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:54:51.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:54:51.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:54:51.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:54:51.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:54:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:51.572+0000 7fc00f1317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:51.576 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:51.572+0000 7fc00f1317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:51.577 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:51.576+0000 7fc00f1317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:51.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:54:51.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:54:51.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:54:51.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:54:51.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:54:51.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:51.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:54:51.991 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:54:51.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:51.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:52.143 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:52.140+0000 7fc00f1317c0 -1 Falling back to public interface 2024-04-25T07:54:52.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:53.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:53.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:53.431 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:54:53.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:54:53.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:53.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:53.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:54.785 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:54:54.780+0000 7fc00f1317c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:54:54.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:54.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:54.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:54:54.855 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:54:54.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:54.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:55.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:56.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:56.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:56.559 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:54:56.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:54:56.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:56.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:56.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:57.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:57.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:57.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:54:57.996 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:54:57.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:57.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:58.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:59.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:59.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:59.420 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:54:59.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:54:59.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:59.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:00.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:01.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:01.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:01.016 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:55:01.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:55:01.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:01.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:01.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:02.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:02.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:02.448 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:55:02.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:55:02.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:02.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:03.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:04.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:04.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:04.023 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:55:04.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:55:04.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:04.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:04.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:05.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:05.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:05.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:55:05.473 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:55:05.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:05.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:06.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:07.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:07.026 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:55:07.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:07.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:55:07.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:07.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:07.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:08.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:08.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:08.473 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:55:08.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:55:08.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:08.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:09.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:10.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:10.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:10.058 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:55:10.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:55:10.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:10.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:10.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:11.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:11.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:11.862 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:55:11.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:55:11.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:11.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:12.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:13.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:13.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:13.661 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:55:13.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:55:13.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:13.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:14.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:15.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:15.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:15.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:55:15.403 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:55:15.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:15.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:16.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:17.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:17.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:17.330 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:55:17.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:55:17.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:17.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:18.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:19.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:19.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:19.565 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:55:19.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:55:19.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:19.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:20.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:21.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:21.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:21.054 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:55:21.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:55:21.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:21.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:55:21.477 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2467567895,v1:127.0.0.1:6803/2467567895] [v2:127.0.0.1:6804/2467567895,v1:127.0.0.1:6805/2467567895] exists,up c6820445-f1b7-4719-b27f-022a76df9a3b 2024-04-25T07:55:21.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:55:21.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:55:21.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:55:21.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2024-04-25T07:55:21.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:55:21.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:55:21.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:55:21.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:55:21.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:55:21.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:55:21.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:55:21.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:55:21.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:55:21.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:55:21.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:55:21.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:55:21.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb40f098-7348-49a3-9f87-6d816e3e0c7e 2024-04-25T07:55:21.482 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 eb40f098-7348-49a3-9f87-6d816e3e0c7e 2024-04-25T07:55:21.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 eb40f098-7348-49a3-9f87-6d816e3e0c7e' 2024-04-25T07:55:21.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:55:21.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpDCpmKOD7HRAAY8O7aelc2ANExU8NTpQ/DQ== 2024-04-25T07:55:21.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpDCpmKOD7HRAAY8O7aelc2ANExU8NTpQ/DQ=="}' 2024-04-25T07:55:21.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb40f098-7348-49a3-9f87-6d816e3e0c7e -i td/osd-markdown/1/new.json 2024-04-25T07:55:21.977 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:55:21.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:55:21.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpDCpmKOD7HRAAY8O7aelc2ANExU8NTpQ/DQ== --osd-uuid eb40f098-7348-49a3-9f87-6d816e3e0c7e 2024-04-25T07:55:22.023 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:22.023+0000 7fef2b4947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:22.032 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:22.031+0000 7fef2b4947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:22.033 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:22.035+0000 7fef2b4947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:22.034 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:22.035+0000 7fef2b4947c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:55:22.034 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:22.035+0000 7fef2b4947c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:55:22.034 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:22.035+0000 7fef2b4947c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:55:22.064 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:22.063+0000 7fef2b4947c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:55:28.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:55:28.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:55:28.975 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:55:28.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:55:28.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:55:29.522 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:55:29.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:55:29.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:55:29.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:55:29.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:55:29.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:55:29.551 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:29.551+0000 7f0a959697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:29.563 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:29.563+0000 7f0a959697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:29.564 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:29.563+0000 7f0a959697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:29.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:55:29.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:55:29.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:55:29.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:55:29.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:55:29.941 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:55:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:55:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:29.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:30.123 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:30.123+0000 7f0a959697c0 -1 Falling back to public interface 2024-04-25T07:55:30.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:31.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:31.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:31.359 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:55:31.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:55:31.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:31.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:31.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:32.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:32.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:32.762 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:55:32.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:55:32.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:32.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:32.883 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:55:32.883+0000 7f0a959697c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:55:33.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:34.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:34.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:34.233 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:55:34.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:55:34.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:34.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:34.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:35.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:35.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:35.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:55:35.701 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:55:35.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:35.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:36.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:37.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:37.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:37.166 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:55:37.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:55:37.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:37.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:37.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:38.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:38.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:38.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:55:38.695 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:55:38.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:38.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:39.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:40.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:40.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:40.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:55:40.530 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:55:40.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:40.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:41.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:42.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:42.154 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:55:42.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:42.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:55:42.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:42.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:42.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:43.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:43.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:43.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:55:43.592 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:55:43.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:43.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:44.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:45.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:45.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:45.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:55:45.191 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:55:45.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:45.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:45.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:46.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:46.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:46.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:55:46.689 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:55:46.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:46.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:47.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:48.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:48.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:48.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:55:48.309 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:55:48.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:48.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:48.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:49.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:49.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:49.909 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:55:49.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:55:49.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:49.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:50.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:51.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:51.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:51.825 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:55:51.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:55:51.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:51.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:52.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:53.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:53.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:53.665 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:55:53.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:55:53.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:53.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:54.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:55.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:55.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:55.495 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:55:55.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:55:55.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:55.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:56.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:55:57.669 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:55:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:58.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:59.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:59.426 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:55:59.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:59.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:55:59.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:59.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:59.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:00.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:00.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:00.941 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T07:56:00.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:56:00.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:00.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:56:01.409 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3973936113,v1:127.0.0.1:6811/3973936113] [v2:127.0.0.1:6812/3973936113,v1:127.0.0.1:6813/3973936113] exists,up eb40f098-7348-49a3-9f87-6d816e3e0c7e 2024-04-25T07:56:01.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:56:01.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:56:01.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:56:01.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2024-04-25T07:56:01.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:56:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:56:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:56:01.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:56:01.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:56:01.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:56:01.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:56:01.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9ae55e63-15e9-422f-858c-632a24fdb9da 2024-04-25T07:56:01.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9ae55e63-15e9-422f-858c-632a24fdb9da' 2024-04-25T07:56:01.416 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 9ae55e63-15e9-422f-858c-632a24fdb9da 2024-04-25T07:56:01.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:56:01.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRDCpmZCusGRAAn8pMu1W+Go7HXa6Vjxgr3g== 2024-04-25T07:56:01.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRDCpmZCusGRAAn8pMu1W+Go7HXa6Vjxgr3g=="}' 2024-04-25T07:56:01.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9ae55e63-15e9-422f-858c-632a24fdb9da -i td/osd-markdown/2/new.json 2024-04-25T07:56:01.929 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:56:01.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:56:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRDCpmZCusGRAAn8pMu1W+Go7HXa6Vjxgr3g== --osd-uuid 9ae55e63-15e9-422f-858c-632a24fdb9da 2024-04-25T07:56:01.970 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:01.971+0000 7f9289f547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:01.972 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:01.971+0000 7f9289f547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:01.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:01.975+0000 7f9289f547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:01.975+0000 7f9289f547c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:56:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:01.975+0000 7f9289f547c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:56:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:01.975+0000 7f9289f547c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:56:02.087 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:02.087+0000 7f9289f547c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:56:08.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:56:08.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:56:08.992 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:56:08.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:56:08.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:56:09.533 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:56:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:56:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:56:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:56:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:56:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:56:09.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:09.563+0000 7fe0685a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:09.564 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:09.563+0000 7fe0685a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:09.566 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:09.567+0000 7fe0685a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:56:09.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:56:09.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:56:09.976 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:56:09.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:56:09.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:56:09.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:56:09.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:09.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:56:09.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:09.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:10.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:10.135+0000 7fe0685a37c0 -1 Falling back to public interface 2024-04-25T07:56:10.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:11.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:11.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:11.407 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:56:11.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:56:11.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:11.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:11.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:12.711 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:56:12.711+0000 7fe0685a37c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:56:12.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:12.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:12.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:56:12.802 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:56:12.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:12.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:13.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:14.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:14.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:14.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:56:14.444 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:56:14.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:14.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:14.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:15.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:15.860 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:56:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:56:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:16.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:17.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:17.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:17.295 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:56:17.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:56:17.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:17.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:17.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:18.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:18.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:18.876 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:56:18.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:56:18.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:18.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:19.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:20.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:20.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:20.289 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:56:20.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:56:20.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:20.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:20.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:21.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:21.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:21.805 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:56:21.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:56:21.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:21.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:22.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:23.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:23.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:23.238 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:56:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:56:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:23.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:23.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:24.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:24.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:24.714 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:56:24.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:56:24.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:24.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:25.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:26.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:26.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:26.157 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:56:26.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:56:26.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:26.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:26.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:27.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:27.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:27.581 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:56:27.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:56:27.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:27.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:28.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:29.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:29.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:29.168 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:56:29.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:56:29.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:29.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:29.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:30.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:30.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:30.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:56:30.834 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:56:30.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:30.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:31.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:32.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:32.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:32.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:56:32.950 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:56:32.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:32.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:33.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:34.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:34.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:34.933 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:56:34.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:56:34.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:34.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:35.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:36.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:36.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:36.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:56:36.852 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:56:36.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:36.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:37.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:38.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:38.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:38.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:56:38.949 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:56:38.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:38.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:39.488 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4167267240,v1:127.0.0.1:6819/4167267240] [v2:127.0.0.1:6820/4167267240,v1:127.0.0.1:6821/4167267240] exists,up 9ae55e63-15e9-422f-858c-632a24fdb9da 2024-04-25T07:56:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:56:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:56:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:56:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2024-04-25T07:56:39.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:56:39.885 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-04-25T07:56:39.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:56:39.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:56:40.450 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' already exists 2024-04-25T07:56:40.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:56:41.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:56:43.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2024-04-25T07:56:43.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2024-04-25T07:56:43.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2024-04-25T07:56:43.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T07:56:43.734 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T07:56:43.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2024-04-25T07:56:43.886 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T07:56:43.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2024-04-25T07:56:43.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-04-25T07:56:43.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2024-04-25T07:56:43.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T07:56:43.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-04-25T07:56:43.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:56:43.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:56:44.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:56:44.428 INFO:tasks.workunit.client.0.smithi102.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:56:44.428 INFO:tasks.workunit.client.0.smithi102.stdout:-1 0.29306 root default 2024-04-25T07:56:44.428 INFO:tasks.workunit.client.0.smithi102.stdout:-3 0.29306 host smithi102 2024-04-25T07:56:44.428 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:44.428 INFO:tasks.workunit.client.0.smithi102.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:56:44.428 INFO:tasks.workunit.client.0.smithi102.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:56:44.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:56:44.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:56:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:56:44.891 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:44.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:56:44.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:56:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:marked down osd.0. 2024-04-25T07:56:45.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:56:55.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:56:55.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:56:55.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:56:56.001 INFO:tasks.workunit.client.0.smithi102.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:56:56.001 INFO:tasks.workunit.client.0.smithi102.stdout:-1 0.29306 root default 2024-04-25T07:56:56.001 INFO:tasks.workunit.client.0.smithi102.stdout:-3 0.29306 host smithi102 2024-04-25T07:56:56.001 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:56.001 INFO:tasks.workunit.client.0.smithi102.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:56:56.001 INFO:tasks.workunit.client.0.smithi102.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:56:56.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:56:56.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:56:56.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:56:56.444 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:56.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:56:56.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:56:56.925 INFO:tasks.workunit.client.0.smithi102.stderr:marked down osd.0. 2024-04-25T07:56:56.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:57:06.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:57:06.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:57:07.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:57:07.512 INFO:tasks.workunit.client.0.smithi102.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:57:07.512 INFO:tasks.workunit.client.0.smithi102.stdout:-1 0.29306 root default 2024-04-25T07:57:07.512 INFO:tasks.workunit.client.0.smithi102.stdout:-3 0.29306 host smithi102 2024-04-25T07:57:07.512 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:57:07.513 INFO:tasks.workunit.client.0.smithi102.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:57:07.513 INFO:tasks.workunit.client.0.smithi102.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:57:07.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:57:07.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:57:07.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:57:07.944 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:57:07.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:57:07.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:57:08.383 INFO:tasks.workunit.client.0.smithi102.stderr:marked down osd.0. 2024-04-25T07:57:08.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:57:18.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:57:18.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:57:18.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:57:18.962 INFO:tasks.workunit.client.0.smithi102.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:57:18.962 INFO:tasks.workunit.client.0.smithi102.stdout:-1 0.29306 root default 2024-04-25T07:57:18.962 INFO:tasks.workunit.client.0.smithi102.stdout:-3 0.29306 host smithi102 2024-04-25T07:57:18.962 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:57:18.962 INFO:tasks.workunit.client.0.smithi102.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:57:18.962 INFO:tasks.workunit.client.0.smithi102.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:57:18.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:57:18.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:57:18.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:57:19.413 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:57:19.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:57:19.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:57:19.923 INFO:tasks.workunit.client.0.smithi102.stderr:marked down osd.0. 2024-04-25T07:57:19.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:57:29.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2024-04-25T07:57:44.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:57:45.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2024-04-25T07:57:45.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2024-04-25T07:57:45.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2024-04-25T07:57:45.539 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:57:45.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T07:57:45.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:57:45.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:57:45.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:57:45.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:57:45.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:57:45.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:57:45.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:57:45.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:57:45.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:57:45.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:57:45.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:57:45.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:57:45.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:57:45.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:57:45.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:57:45.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:57:45.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:57:45.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:57:45.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:57:45.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:57:45.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:57:45.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:57:45.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:57:45.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:45.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:57:45.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178198 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:57:45.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:57:45.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:57:45.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:57:45.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:57:45.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:57:45.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:57:45.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:57:45.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:57:45.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:57:45.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:57:45.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:57:45.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:57:45.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:57:45.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:57:45.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:57:45.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:57:45.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:45.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:57:45.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178198 2024-04-25T07:57:45.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:57:45.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:57:45.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:57:45.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:57:45.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:45.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:57:45.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.178198 2024-04-25T07:57:45.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:57:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:57:46.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:57:46.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:57:46.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:57:46.173 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:57:46.174 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:46.174 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:57:46.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:57:46.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:57:46.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:57:46.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:57:46.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:57:46.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:57:46.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:57:46.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:57:46.411 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:57:46.411 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:57:46.411 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:57:46.412 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:57:46.412 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:46.412 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:57:46.412 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178198/ceph-mon.a.asok 2024-04-25T07:57:46.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:57:46.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178198/ceph-mon.a.asok config get fsid 2024-04-25T07:57:46.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:57:46.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:57:46.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:57:46.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:57:46.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:57:46.527 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:57:46.527 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:57:46.527 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:57:46.527 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:57:46.527 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:46.528 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:57:46.528 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178198/ceph-mon.a.asok 2024-04-25T07:57:46.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:57:46.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178198/ceph-mon.a.asok config get mon_host 2024-04-25T07:57:46.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2024-04-25T07:57:46.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:57:46.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:57:46.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:57:46.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:57:46.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:57:46.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:57:47.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:57:47.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:57:47.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:57:47.547 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:57:47.547 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:47.547 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:57:47.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:57:47.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:57:47.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:57:47.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2024-04-25T07:57:47.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:57:47.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:57:47.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:57:47.587 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:57:47.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:57:47.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:57:47.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1534fa0f-b9e6-40d7-b8e4-c3fe3af5226f 2024-04-25T07:57:47.589 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 1534fa0f-b9e6-40d7-b8e4-c3fe3af5226f 2024-04-25T07:57:47.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1534fa0f-b9e6-40d7-b8e4-c3fe3af5226f' 2024-04-25T07:57:47.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:57:47.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7DCpm2v8YJBAA5lmkR1RAY+xdDvMco69mDQ== 2024-04-25T07:57:47.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7DCpm2v8YJBAA5lmkR1RAY+xdDvMco69mDQ=="}' 2024-04-25T07:57:47.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1534fa0f-b9e6-40d7-b8e4-c3fe3af5226f -i td/osd-markdown/0/new.json 2024-04-25T07:57:47.937 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:57:47.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:57:47.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7DCpm2v8YJBAA5lmkR1RAY+xdDvMco69mDQ== --osd-uuid 1534fa0f-b9e6-40d7-b8e4-c3fe3af5226f 2024-04-25T07:57:47.984 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:47.981+0000 7f79ff9ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:48.015 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:48.013+0000 7f79ff9ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:48.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:48.013+0000 7f79ff9ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:48.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:48.013+0000 7f79ff9ea7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:57:48.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:48.013+0000 7f79ff9ea7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:57:48.017 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:48.013+0000 7f79ff9ea7c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:57:48.054 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:48.053+0000 7f79ff9ea7c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:57:54.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:57:54.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:57:54.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:57:54.505 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T07:57:54.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:57:55.056 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T07:57:55.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:57:55.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:57:55.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:57:55.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:57:55.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:57:55.085 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:55.081+0000 7f07210947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:55.087 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:55.085+0000 7f07210947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:55.088 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:55.085+0000 7f07210947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:55.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:57:55.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:57:55.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:57:55.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:57:55.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:57:55.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:55.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:57:55.506 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:57:55.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:55.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:55.654 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:55.653+0000 7f07210947c0 -1 Falling back to public interface 2024-04-25T07:57:55.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:56.935 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:57:56.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:56.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:56.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:57:56.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:56.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:57.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:58.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:58.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:58.352 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:57:58.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:57:58.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:58.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:58.403 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:57:58.401+0000 7f07210947c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:57:58.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:59.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:59.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:59.935 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:57:59.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:57:59.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:59.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:00.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:01.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:01.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:01.384 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:58:01.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:58:01.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:01.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:01.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:02.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:02.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:02.795 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:58:02.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:58:02.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:02.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:03.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:04.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:04.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:04.314 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:58:04.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:58:04.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:04.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:05.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:05.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:05.751 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:58:05.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:58:05.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:05.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:06.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:07.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:07.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:07.278 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:58:07.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:58:07.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:07.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:07.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:08.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:08.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:08.752 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:58:08.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:58:08.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:08.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:09.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:10.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:10.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:10.280 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:58:10.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:58:10.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:10.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:10.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:11.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:11.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:11.712 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:58:11.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:58:11.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:11.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:12.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:13.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:13.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:13.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:58:13.266 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:58:13.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:13.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:14.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:15.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:15.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:15.149 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:58:15.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:58:15.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:15.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:15.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:16.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:16.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:16.763 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:58:16.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:58:16.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:16.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:18.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:19.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:19.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:19.159 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:58:19.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:58:19.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:19.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:20.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:21.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:21.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:21.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:58:21.088 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:58:21.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:21.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:21.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:22.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:22.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:22.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:58:22.678 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:58:22.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:22.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:23.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:24.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:24.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:24.293 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:58:24.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:58:24.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:24.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:58:24.698 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3501047393,v1:127.0.0.1:6803/3501047393] [v2:127.0.0.1:6804/3501047393,v1:127.0.0.1:6805/3501047393] exists,up 1534fa0f-b9e6-40d7-b8e4-c3fe3af5226f 2024-04-25T07:58:24.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:58:24.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:58:24.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:58:24.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2024-04-25T07:58:24.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:58:24.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:58:24.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:58:24.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:58:24.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:58:24.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:58:24.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:58:24.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:58:24.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:58:24.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4b0bc3ae-ff30-410b-9f54-04c8a4010350 2024-04-25T07:58:24.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4b0bc3ae-ff30-410b-9f54-04c8a4010350' 2024-04-25T07:58:24.706 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 4b0bc3ae-ff30-410b-9f54-04c8a4010350 2024-04-25T07:58:24.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:58:24.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAgDSpmeMryKhAAUYdFRwryUL4Ho+eiaNnbdQ== 2024-04-25T07:58:24.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAgDSpmeMryKhAAUYdFRwryUL4Ho+eiaNnbdQ=="}' 2024-04-25T07:58:24.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4b0bc3ae-ff30-410b-9f54-04c8a4010350 -i td/osd-markdown/1/new.json 2024-04-25T07:58:25.335 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:58:25.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:58:25.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgDSpmeMryKhAAUYdFRwryUL4Ho+eiaNnbdQ== --osd-uuid 4b0bc3ae-ff30-410b-9f54-04c8a4010350 2024-04-25T07:58:25.379 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:25.376+0000 7feef63847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:25.412 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:25.408+0000 7feef63847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:25.414 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:25.412+0000 7feef63847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:25.414 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:25.412+0000 7feef63847c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:58:25.414 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:25.412+0000 7feef63847c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:58:25.415 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:25.412+0000 7feef63847c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:58:25.451 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:25.448+0000 7feef63847c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:58:32.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:58:32.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:58:32.200 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T07:58:32.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:58:32.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:58:32.752 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T07:58:32.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:58:32.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:58:32.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:58:32.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:58:32.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:58:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:32.780+0000 7f3b3bc4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:32.784 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:32.780+0000 7f3b3bc4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:32.786 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:32.784+0000 7f3b3bc4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:33.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:58:33.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:58:33.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:58:33.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:58:33.197 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:58:33.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:58:33.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:33.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:58:33.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:33.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:33.338 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:33.336+0000 7f3b3bc4e7c0 -1 Falling back to public interface 2024-04-25T07:58:33.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:34.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:34.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:34.626 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:58:34.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:58:34.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:34.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:35.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:36.023 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:58:36.020+0000 7f3b3bc4e7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:58:36.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:36.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:36.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:58:36.050 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:58:36.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:36.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:36.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:37.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:37.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:37.607 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:58:37.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:58:37.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:37.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:38.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:39.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:39.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:39.100 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:58:39.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:58:39.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:39.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:39.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:40.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:40.566 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:58:40.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:40.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:58:40.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:40.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:41.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:42.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:42.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:42.217 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:58:42.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:58:42.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:42.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:42.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:43.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:43.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:43.643 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:58:43.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:58:43.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:43.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:44.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:45.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:45.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:45.141 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:58:45.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:58:45.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:45.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:45.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:46.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:46.547 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:58:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:58:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:47.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:48.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:48.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:48.094 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:58:48.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:58:48.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:48.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:48.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:49.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:49.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:49.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:58:49.521 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:58:49.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:49.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:50.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:51.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:51.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:51.218 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:58:51.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:58:51.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:51.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:52.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:53.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:53.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:53.049 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:58:53.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:58:53.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:53.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:53.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:54.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:54.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:54.856 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:58:54.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:58:54.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:54.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:55.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:56.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:56.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:56.739 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:58:56.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:58:56.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:56.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:58.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:58.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:58.632 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:58:58.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:58:58.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:58.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:59.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:00.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:00.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:00.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:59:00.104 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:59:00.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:00.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:59:00.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:01.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:01.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:01.967 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:59:01.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:59:01.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:01.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:59:02.497 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/28487479,v1:127.0.0.1:6811/28487479] [v2:127.0.0.1:6812/28487479,v1:127.0.0.1:6813/28487479] exists,up 4b0bc3ae-ff30-410b-9f54-04c8a4010350 2024-04-25T07:59:02.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:59:02.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:59:02.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:59:02.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2024-04-25T07:59:02.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:59:02.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:59:02.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:59:02.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:59:02.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:59:02.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:59:02.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:59:02.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:59:02.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:59:02.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:59:02.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a28f6991-b336-4aff-a4cb-14c02dd46b3b 2024-04-25T07:59:02.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a28f6991-b336-4aff-a4cb-14c02dd46b3b' 2024-04-25T07:59:02.505 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 a28f6991-b336-4aff-a4cb-14c02dd46b3b 2024-04-25T07:59:02.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:59:02.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGDSpmFC3sHhAAp0f9jwAt8mLWHfQX8AybUg== 2024-04-25T07:59:02.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGDSpmFC3sHhAAp0f9jwAt8mLWHfQX8AybUg=="}' 2024-04-25T07:59:02.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a28f6991-b336-4aff-a4cb-14c02dd46b3b -i td/osd-markdown/2/new.json 2024-04-25T07:59:03.004 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:59:03.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:59:03.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGDSpmFC3sHhAAp0f9jwAt8mLWHfQX8AybUg== --osd-uuid a28f6991-b336-4aff-a4cb-14c02dd46b3b 2024-04-25T07:59:03.050 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:03.051+0000 7ffb202757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:03.082 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:03.083+0000 7ffb202757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:03.084 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:03.083+0000 7ffb202757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:03.084 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:03.083+0000 7ffb202757c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:59:03.085 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:03.083+0000 7ffb202757c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:59:03.085 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:03.083+0000 7ffb202757c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:59:03.121 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:03.123+0000 7ffb202757c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:59:10.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:59:10.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:59:10.169 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T07:59:10.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:59:10.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:59:10.704 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T07:59:10.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:59:10.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:59:10.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:59:10.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:59:10.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:59:10.733 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:10.735+0000 7f278df0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:10.735 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:10.735+0000 7f278df0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:10.737 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:10.735+0000 7f278df0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:59:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:59:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:59:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:59:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:59:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:59:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:11.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:59:11.119 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T07:59:11.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:11.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:11.306 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:11.307+0000 7f278df0e7c0 -1 Falling back to public interface 2024-04-25T07:59:11.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:12.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:12.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:12.533 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T07:59:12.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:59:12.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:12.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:13.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:13.895 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T07:59:13.895+0000 7f278df0e7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:59:14.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:14.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:14.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:59:14.005 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T07:59:14.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:14.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:14.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:15.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:15.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:15.719 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T07:59:15.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:59:15.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:15.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:16.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:17.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:17.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:17.146 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T07:59:17.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:59:17.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:17.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:17.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:18.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:18.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:18.620 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T07:59:18.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:59:18.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:18.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:19.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:20.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:20.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:20.347 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T07:59:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:59:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:20.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:20.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:21.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:21.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:21.868 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T07:59:21.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:59:21.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:21.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:22.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:23.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:23.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:59:23.277 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T07:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:23.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:23.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:24.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:24.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:24.843 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T07:59:24.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:59:24.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:24.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:25.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:26.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:26.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:26.447 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T07:59:26.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:59:26.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:26.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:26.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:27.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:27.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:27.873 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T07:59:27.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:59:27.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:27.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:28.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:29.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:29.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:29.287 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T07:59:29.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:59:29.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:29.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:29.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:30.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:30.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:30.928 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T07:59:30.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:59:30.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:30.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:32.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:33.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:33.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:33.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:59:33.103 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T07:59:33.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:33.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:33.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:34.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:34.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:34.941 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T07:59:34.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:59:34.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:34.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:35.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:36.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:36.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:36.863 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T07:59:36.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:59:36.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:36.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:37.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:38.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:38.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:38.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:59:38.718 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T07:59:38.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:38.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:39.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:40.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:40.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:40.134 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T07:59:40.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:59:40.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:40.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:40.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:41.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:41.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:41.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:59:41.596 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T07:59:41.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:41.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:42.027 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3072070453,v1:127.0.0.1:6819/3072070453] [v2:127.0.0.1:6820/3072070453,v1:127.0.0.1:6821/3072070453] exists,up a28f6991-b336-4aff-a4cb-14c02dd46b3b 2024-04-25T07:59:42.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:59:42.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:59:42.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:59:42.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2024-04-25T07:59:42.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:59:42.449 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-04-25T07:59:42.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:59:42.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:59:43.087 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' already exists 2024-04-25T07:59:43.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:59:44.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:59:46.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2024-04-25T07:59:46.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2024-04-25T07:59:46.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2024-04-25T07:59:46.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T07:59:47.135 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T07:59:47.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2024-04-25T07:59:47.294 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T07:59:47.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2024-04-25T07:59:47.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-04-25T07:59:47.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2024-04-25T07:59:47.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T07:59:47.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-04-25T07:59:47.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:59:47.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:59:47.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:59:47.830 INFO:tasks.workunit.client.0.smithi102.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:59:47.830 INFO:tasks.workunit.client.0.smithi102.stdout:-1 0.29306 root default 2024-04-25T07:59:47.830 INFO:tasks.workunit.client.0.smithi102.stdout:-3 0.29306 host smithi102 2024-04-25T07:59:47.831 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:59:47.831 INFO:tasks.workunit.client.0.smithi102.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:59:47.831 INFO:tasks.workunit.client.0.smithi102.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:59:47.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:59:47.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:59:47.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:59:48.298 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:59:48.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:59:48.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:59:48.711 INFO:tasks.workunit.client.0.smithi102.stderr:marked down osd.0. 2024-04-25T07:59:48.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:59:58.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:59:58.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:59:58.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:59:59.268 INFO:tasks.workunit.client.0.smithi102.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:59:59.268 INFO:tasks.workunit.client.0.smithi102.stdout:-1 0.29306 root default 2024-04-25T07:59:59.268 INFO:tasks.workunit.client.0.smithi102.stdout:-3 0.29306 host smithi102 2024-04-25T07:59:59.269 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:59:59.269 INFO:tasks.workunit.client.0.smithi102.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:59:59.269 INFO:tasks.workunit.client.0.smithi102.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:59:59.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:59:59.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:59:59.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:59:59.730 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:59:59.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:59:59.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T08:00:00.159 INFO:tasks.workunit.client.0.smithi102.stderr:marked down osd.0. 2024-04-25T08:00:00.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T08:00:10.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T08:00:10.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T08:00:10.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T08:00:10.720 INFO:tasks.workunit.client.0.smithi102.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T08:00:10.720 INFO:tasks.workunit.client.0.smithi102.stdout:-1 0.29306 root default 2024-04-25T08:00:10.720 INFO:tasks.workunit.client.0.smithi102.stdout:-3 0.29306 host smithi102 2024-04-25T08:00:10.720 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:00:10.720 INFO:tasks.workunit.client.0.smithi102.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T08:00:10.720 INFO:tasks.workunit.client.0.smithi102.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T08:00:10.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T08:00:10.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T08:00:10.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T08:00:11.162 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:00:11.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T08:00:11.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T08:00:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:marked down osd.0. 2024-04-25T08:00:11.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T08:00:21.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T08:00:21.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T08:00:21.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T08:00:22.154 INFO:tasks.workunit.client.0.smithi102.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T08:00:22.154 INFO:tasks.workunit.client.0.smithi102.stdout:-1 0.29306 root default 2024-04-25T08:00:22.155 INFO:tasks.workunit.client.0.smithi102.stdout:-3 0.29306 host smithi102 2024-04-25T08:00:22.155 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:00:22.155 INFO:tasks.workunit.client.0.smithi102.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T08:00:22.155 INFO:tasks.workunit.client.0.smithi102.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T08:00:22.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T08:00:22.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T08:00:22.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T08:00:22.571 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:00:22.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T08:00:22.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T08:00:23.022 INFO:tasks.workunit.client.0.smithi102.stderr:marked down osd.0. 2024-04-25T08:00:23.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T08:00:33.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2024-04-25T08:00:33.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2024-04-25T08:00:33.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2024-04-25T08:00:33.466 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-04-25T08:00:33.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T08:00:33.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T08:00:33.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:00:33.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T08:00:33.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:00:33.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:00:33.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:00:33.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:00:33.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:00:33.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:00:33.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:00:33.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:00:33.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:00:33.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:00:33.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:00:33.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:00:33.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:33.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:00:33.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:00:33.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:33.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:00:33.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:00:33.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T08:00:33.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:00:33.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:33.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:00:33.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178198 2024-04-25T08:00:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:00:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:00:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T08:00:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T08:00:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T08:00:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T08:00:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T08:00:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:00:33.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T08:00:33.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:00:33.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:00:33.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:00:33.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:00:33.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:00:33.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:00:33.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:00:33.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:00:33.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:00:33.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:00:33.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:00:33.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:00:33.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:33.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:00:33.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:00:33.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:33.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:00:33.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:00:33.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T08:00:33.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:00:33.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:33.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:00:33.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178198 2024-04-25T08:00:33.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:00:33.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:00:33.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T08:00:33.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:00:33.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:33.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:00:33.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.178198 2024-04-25T08:00:33.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:00:33.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:00:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:00:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T08:00:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2024-04-25T08:00:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2024-04-25T08:00:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2024-04-25T08:00:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T08:00:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:00:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:00:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:00:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T08:00:33.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T08:00:34.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:00:34.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:00:34.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:00:34.298 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:00:34.298 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:34.298 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:00:34.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T08:00:34.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:00:34.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:00:34.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:00:34.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:00:34.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:00:34.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:00:34.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:00:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:00:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:00:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:00:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:00:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:00:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178198/ceph-mon.a.asok 2024-04-25T08:00:34.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:00:34.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178198/ceph-mon.a.asok config get fsid 2024-04-25T08:00:34.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:00:34.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:00:34.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:00:34.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:00:34.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:00:34.648 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:00:34.648 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:00:34.648 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:00:34.648 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:00:34.648 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:34.648 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:00:34.648 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178198/ceph-mon.a.asok 2024-04-25T08:00:34.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:00:34.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178198/ceph-mon.a.asok config get mon_host 2024-04-25T08:00:34.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2024-04-25T08:00:34.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T08:00:34.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:00:34.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:00:34.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:00:34.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T08:00:34.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:00:35.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:00:35.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:00:35.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:00:35.675 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:00:35.675 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:35.675 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:00:35.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T08:00:35.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:00:35.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:00:35.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2024-04-25T08:00:35.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T08:00:35.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:00:35.711 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:00:35.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:00:35.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T08:00:35.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:00:35.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=832aef4c-e6a8-4f91-8005-8e5f999e04aa 2024-04-25T08:00:35.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 832aef4c-e6a8-4f91-8005-8e5f999e04aa' 2024-04-25T08:00:35.714 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 832aef4c-e6a8-4f91-8005-8e5f999e04aa 2024-04-25T08:00:35.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:00:35.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCjDSpmUYGIKxAADULGNZwtGPe8UvRrp8ZVYA== 2024-04-25T08:00:35.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCjDSpmUYGIKxAADULGNZwtGPe8UvRrp8ZVYA=="}' 2024-04-25T08:00:35.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 832aef4c-e6a8-4f91-8005-8e5f999e04aa -i td/osd-markdown/0/new.json 2024-04-25T08:00:36.054 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:00:36.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T08:00:36.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjDSpmUYGIKxAADULGNZwtGPe8UvRrp8ZVYA== --osd-uuid 832aef4c-e6a8-4f91-8005-8e5f999e04aa 2024-04-25T08:00:36.107 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:36.105+0000 7fd6b3b227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:36.131 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:36.129+0000 7fd6b3b227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:36.133 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:36.129+0000 7fd6b3b227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:36.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:36.133+0000 7fd6b3b227c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T08:00:36.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:36.133+0000 7fd6b3b227c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T08:00:36.134 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:36.133+0000 7fd6b3b227c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T08:00:36.171 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:36.169+0000 7fd6b3b227c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T08:00:42.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T08:00:42.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:00:42.685 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:00:42.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:00:42.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:00:43.214 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:00:43.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:00:43.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:00:43.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:00:43.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:00:43.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:00:43.243 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:43.241+0000 7fc4beb877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:43.246 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:43.245+0000 7fc4beb877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:43.247 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:43.245+0000 7fc4beb877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:43.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:00:43.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:00:43.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:00:43.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:00:43.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:00:43.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:43.671 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:00:43.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:00:43.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:43.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:43.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:43.821+0000 7fc4beb877c0 -1 Falling back to public interface 2024-04-25T08:00:44.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:45.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:45.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:45.086 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:00:45.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:00:45.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:45.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:45.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:46.430 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:00:46.429+0000 7fc4beb877c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:00:46.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:46.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:46.491 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:00:46.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:00:46.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:46.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:47.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:48.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:48.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:48.171 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:00:48.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:00:48.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:48.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:48.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:49.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:49.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:49.630 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:00:49.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:00:49.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:49.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:50.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:51.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:51.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:51.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:00:51.059 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:00:51.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:51.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:51.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:52.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:52.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:52.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:00:52.631 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:00:52.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:52.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:53.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:54.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:54.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:54.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:00:54.054 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:00:54.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:54.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:54.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:55.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:55.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:55.604 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:00:55.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:00:55.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:55.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:56.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:57.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:57.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:57.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:00:57.020 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:00:57.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:57.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:57.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:58.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:58.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:58.439 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:00:58.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:00:58.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:58.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:58.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:59.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:59.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:59.900 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:00:59.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:00:59.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:59.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:00.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:01.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:01.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:01.649 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:01:01.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:01:01.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:01.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:02.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:03.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:03.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:03.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:01:03.909 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:01:03.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:03.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:04.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:05.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:05.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:05.343 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:01:05.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:01:05.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:05.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:06.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:01:07.176 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:01:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:07.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:08.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:09.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:09.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:09.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:01:09.197 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:01:09.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:09.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:10.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:11.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:11.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:11.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:01:11.205 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:01:11.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:11.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:11.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:12.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:12.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:12.867 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:01:12.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:01:12.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:12.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:13.327 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/413951196,v1:127.0.0.1:6803/413951196] [v2:127.0.0.1:6804/413951196,v1:127.0.0.1:6805/413951196] exists,up 832aef4c-e6a8-4f91-8005-8e5f999e04aa 2024-04-25T08:01:13.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:01:13.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:01:13.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:01:13.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2024-04-25T08:01:13.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T08:01:13.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T08:01:13.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:01:13.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:01:13.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:01:13.331 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:01:13.331 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:01:13.331 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:01:13.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T08:01:13.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T08:01:13.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:01:13.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:01:13.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9bfa3649-6636-4856-b937-5f341dd392ec 2024-04-25T08:01:13.333 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 9bfa3649-6636-4856-b937-5f341dd392ec 2024-04-25T08:01:13.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9bfa3649-6636-4856-b937-5f341dd392ec' 2024-04-25T08:01:13.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:01:13.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJDSpmT3wuFRAAsuTOkwpsGwFOaVHitJI25w== 2024-04-25T08:01:13.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJDSpmT3wuFRAAsuTOkwpsGwFOaVHitJI25w=="}' 2024-04-25T08:01:13.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9bfa3649-6636-4856-b937-5f341dd392ec -i td/osd-markdown/1/new.json 2024-04-25T08:01:13.814 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:01:13.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T08:01:13.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJDSpmT3wuFRAAsuTOkwpsGwFOaVHitJI25w== --osd-uuid 9bfa3649-6636-4856-b937-5f341dd392ec 2024-04-25T08:01:13.865 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:13.860+0000 7fb939a027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:13.881 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:13.876+0000 7fb939a027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:13.882 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:13.880+0000 7fb939a027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:13.883 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:13.880+0000 7fb939a027c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T08:01:13.883 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:13.880+0000 7fb939a027c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T08:01:13.883 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:13.880+0000 7fb939a027c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T08:01:13.920 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:13.916+0000 7fb939a027c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T08:01:20.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T08:01:20.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:01:20.661 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T08:01:20.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:01:20.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:01:21.246 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:01:21.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:01:21.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:01:21.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:01:21.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:01:21.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:01:21.277 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:21.272+0000 7f26929217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:21.279 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:21.276+0000 7f26929217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:21.280 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:21.276+0000 7f26929217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:21.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:01:21.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:01:21.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:01:21.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:01:21.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:01:21.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:21.735 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:01:21.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:01:21.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:21.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:21.859 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:21.856+0000 7f26929217c0 -1 Falling back to public interface 2024-04-25T08:01:22.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:23.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:23.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:23.150 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:01:23.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:01:23.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:23.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:23.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:24.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:24.557 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:01:24.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:24.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:01:24.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:24.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:24.558 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:24.556+0000 7f26929217c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:01:25.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:26.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:26.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:26.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:01:26.118 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:01:26.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:26.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:26.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:27.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:27.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:27.651 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:01:27.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:01:27.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:27.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:28.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:29.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:29.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:29.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:01:29.101 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:01:29.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:29.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:29.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:30.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:30.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:30.576 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:01:30.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:01:30.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:30.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:31.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:32.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:32.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:32.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:01:32.094 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:01:32.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:32.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:32.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:33.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:33.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:33.718 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:01:33.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:01:33.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:33.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:34.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:35.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:35.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:35.134 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:01:35.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:01:35.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:35.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:35.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:36.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:36.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:36.694 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:01:36.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:01:36.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:36.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:37.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:38.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:38.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:38.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:01:38.243 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:01:38.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:38.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:38.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:39.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:39.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:39.827 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:01:39.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:01:39.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:39.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:40.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:41.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:41.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:41.398 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:01:41.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:01:41.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:41.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:42.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:43.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:43.210 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:01:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:01:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:43.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:44.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:45.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:45.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:45.084 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:01:45.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:01:45.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:45.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:46.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:47.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:47.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:47.387 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:01:47.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:01:47.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:47.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:48.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:49.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:49.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:49.215 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:01:49.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:01:49.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:49.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:50.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:51.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:51.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:51.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:01:51.061 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:01:51.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:51.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:51.515 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3701183688,v1:127.0.0.1:6811/3701183688] [v2:127.0.0.1:6812/3701183688,v1:127.0.0.1:6813/3701183688] exists,up 9bfa3649-6636-4856-b937-5f341dd392ec 2024-04-25T08:01:51.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:01:51.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:01:51.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:01:51.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2024-04-25T08:01:51.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T08:01:51.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:01:51.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:01:51.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:01:51.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:01:51.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:01:51.519 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:01:51.519 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:01:51.519 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:01:51.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T08:01:51.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' 2024-04-25T08:01:51.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:01:51.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:01:51.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:01:51.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T08:01:51.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T08:01:51.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:01:51.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:01:51.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:01:51.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:01:51.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:01:51.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:01:51.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T08:01:51.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:01:51.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4cc60679-88a3-4772-bbbb-0c94df80895f 2024-04-25T08:01:51.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4cc60679-88a3-4772-bbbb-0c94df80895f' 2024-04-25T08:01:51.522 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 4cc60679-88a3-4772-bbbb-0c94df80895f 2024-04-25T08:01:51.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:01:51.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvDSpmRosJIBAAkRRfzHMvIEWX8WmupA70Jw== 2024-04-25T08:01:51.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvDSpmRosJIBAAkRRfzHMvIEWX8WmupA70Jw=="}' 2024-04-25T08:01:51.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4cc60679-88a3-4772-bbbb-0c94df80895f -i td/osd-markdown/2/new.json 2024-04-25T08:01:52.043 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:01:52.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T08:01:52.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvDSpmRosJIBAAkRRfzHMvIEWX8WmupA70Jw== --osd-uuid 4cc60679-88a3-4772-bbbb-0c94df80895f 2024-04-25T08:01:52.093 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:52.092+0000 7fc7dcc6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:52.120 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:52.116+0000 7fc7dcc6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:52.121 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:52.116+0000 7fc7dcc6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:52.122 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:52.120+0000 7fc7dcc6e7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T08:01:52.122 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:52.120+0000 7fc7dcc6e7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T08:01:52.122 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:52.120+0000 7fc7dcc6e7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T08:01:52.159 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:52.156+0000 7fc7dcc6e7c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T08:01:58.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T08:01:58.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:01:58.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:01:58.887 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T08:01:58.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:01:59.448 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T08:01:59.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:01:59.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:01:59.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=df174043-be53-41dd-8457-78a3d5fb1c9b --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178198/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:01:59.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:01:59.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:01:59.477 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:59.480+0000 7f87a3c7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:59.520+0000 7f87a3c7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:59.519 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:01:59.520+0000 7f87a3c7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:59.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:01:59.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:01:59.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:01:59.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:01:59.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:01:59.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:59.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:01:59.898 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:01:59.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:59.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:00.087 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:00.088+0000 7f87a3c7b7c0 -1 Falling back to public interface 2024-04-25T08:02:00.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:01.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:01.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:01.310 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:02:01.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:02:01.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:01.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:01.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:02.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:02.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:02.725 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:02:02.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:02:02.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:02.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:02.796 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:02.795+0000 7f87a3c7b7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:02:03.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:04.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:04.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:04.246 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:02:04.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:02:04.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:04.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:04.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:05.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:05.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:05.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:02:05.842 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:02:05.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:05.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:06.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:07.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:07.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:07.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:02:07.290 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:02:07.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:07.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:07.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:08.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:08.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:08.861 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:02:08.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:02:08.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:08.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:09.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:10.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:10.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:10.292 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:02:10.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:02:10.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:10.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:10.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:11.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:11.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:11.800 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:02:11.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:02:11.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:11.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:12.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:13.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:13.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:13.239 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:02:13.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:02:13.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:13.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:13.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:14.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:14.810 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:02:14.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:14.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:02:14.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:14.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:15.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:16.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:16.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:16.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:02:16.324 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:02:16.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:16.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:16.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:17.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:17.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:17.828 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:02:17.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:02:17.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:17.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:18.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:19.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:19.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:19.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:02:19.566 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:02:19.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:19.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:20.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:21.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:21.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:21.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:02:21.737 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:02:21.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:21.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:22.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:23.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:23.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:23.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:02:23.618 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:02:23.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:23.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:24.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:25.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:25.445 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:02:25.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:25.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:02:25.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:25.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:26.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:27.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:27.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:27.140 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:02:27.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:02:27.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:27.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:28.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:29.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:29.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:29.068 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:02:29.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:02:29.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:29.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:29.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:30.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:30.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:30.489 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:02:30.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:02:30.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:30.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:30.919 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/871070499,v1:127.0.0.1:6819/871070499] [v2:127.0.0.1:6820/871070499,v1:127.0.0.1:6821/871070499] exists,up 4cc60679-88a3-4772-bbbb-0c94df80895f 2024-04-25T08:02:30.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:02:30.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:02:30.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:02:30.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2024-04-25T08:02:30.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=197558 2024-04-25T08:02:30.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 197558 2024-04-25T08:02:30.929 INFO:tasks.workunit.client.0.smithi102.stdout: PID TTY TIME CMD 2024-04-25T08:02:30.929 INFO:tasks.workunit.client.0.smithi102.stdout: 197558 ? 00:00:00 ceph-osd 2024-04-25T08:02:30.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2024-04-25T08:02:30.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2024-04-25T08:02:30.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2024-04-25T08:02:31.320 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:02:31.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2024-04-25T08:02:31.989 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already stopped. 2024-04-25T08:02:32.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2024-04-25T08:02:47.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2024-04-25T08:02:47.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2024-04-25T08:02:47.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2024-04-25T08:02:47.420 INFO:tasks.workunit.client.0.smithi102.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-04-25T08:02:47.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 197558 2024-04-25T08:02:47.429 INFO:tasks.workunit.client.0.smithi102.stdout: PID TTY TIME CMD 2024-04-25T08:02:47.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T08:02:47.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T08:02:47.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:02:47.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T08:02:47.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:02:47.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:02:47.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:02:47.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:02:47.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:02:47.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:02:47.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:02:47.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:02:47.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:02:47.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:02:47.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:02:47.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:02:47.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:02:47.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:02:47.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:02:47.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:02:47.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:02:47.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:02:47.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T08:02:47.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:02:47.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:02:47.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:02:47.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178198 2024-04-25T08:02:47.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:02:47.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:02:47.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:02:47.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-markdown 0 2024-04-25T08:02:47.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T08:02:47.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:02:47.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T08:02:47.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:02:47.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:02:47.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:02:47.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:02:47.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:02:47.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:02:47.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:02:47.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:02:47.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:02:47.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:02:47.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:02:47.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:02:47.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:02:47.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:02:47.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:02:47.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:02:47.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:02:47.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:02:47.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T08:02:47.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:02:47.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:02:47.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178198 2024-04-25T08:02:47.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178198 2024-04-25T08:02:47.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:02:47.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:02:47.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:02:47.784 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:02:47.785 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:02:47.797 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2024-04-25T08:02:47.797 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2024-04-25T08:02:47.853 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:02:47.858 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:02:47.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:02:47.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-reactivate 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2024-04-25T08:02:47.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2024-04-25T08:02:47.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=ee6a59fa-be7f-4597-87c1-bd567473cca1 --auth-supported=none ' 2024-04-25T08:02:47.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2024-04-25T08:02:47.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2024-04-25T08:02:47.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:02:47.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2024-04-25T08:02:47.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2024-04-25T08:02:47.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2024-04-25T08:02:47.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2024-04-25T08:02:47.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2024-04-25T08:02:47.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T08:02:47.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:02:47.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T08:02:47.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:02:47.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:02:47.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:02:47.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:02:47.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:02:47.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:02:47.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:02:47.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:02:47.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:02:47.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:02:47.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:02:47.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:02:47.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:02:47.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:02:47.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:02:47.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:02:47.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:02:47.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:02:47.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T08:02:47.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:02:47.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:02:47.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202171 2024-04-25T08:02:47.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202171 2024-04-25T08:02:47.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:02:47.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:02:47.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2024-04-25T08:02:47.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:02:47.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:02:47.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202171 2024-04-25T08:02:47.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.202171 2024-04-25T08:02:47.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:02:47.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2024-04-25T08:02:47.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2024-04-25T08:02:48.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:02:48.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:02:48.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:02:48.458 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:02:48.458 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:02:48.458 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202171 2024-04-25T08:02:48.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202171/$cluster-$name.asok' 2024-04-25T08:02:48.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.202171/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:02:48.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:02:48.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:02:48.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:02:48.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:02:48.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:02:48.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:02:48.688 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:02:48.688 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:02:48.688 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:02:48.688 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:02:48.688 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:02:48.688 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202171 2024-04-25T08:02:48.688 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202171/ceph-mon.a.asok 2024-04-25T08:02:48.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:02:48.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.202171/ceph-mon.a.asok config get fsid 2024-04-25T08:02:48.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:02:48.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:02:48.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:02:48.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:02:48.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:02:48.807 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:02:48.807 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:02:48.807 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:02:48.807 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:02:48.807 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:02:48.807 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202171 2024-04-25T08:02:48.807 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.202171/ceph-mon.a.asok 2024-04-25T08:02:48.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:02:48.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.202171/ceph-mon.a.asok config get mon_host 2024-04-25T08:02:48.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2024-04-25T08:02:48.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2024-04-25T08:02:48.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:02:48.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:02:48.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:02:48.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2024-04-25T08:02:48.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:02:49.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:02:49.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:02:49.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:02:49.832 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:02:49.832 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:02:49.832 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202171 2024-04-25T08:02:49.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202171/$cluster-$name.asok' 2024-04-25T08:02:49.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:02:49.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.202171/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:02:49.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2024-04-25T08:02:49.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2024-04-25T08:02:49.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:02:49.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:02:49.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=ee6a59fa-be7f-4597-87c1-bd567473cca1 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:02:49.869 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:02:49.870 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202171 2024-04-25T08:02:49.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202171/$cluster-$name.asok' 2024-04-25T08:02:49.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202171/$cluster-$name.asok' 2024-04-25T08:02:49.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:02:49.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:02:49.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:02:49.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-04-25T08:02:49.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-04-25T08:02:49.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:02:49.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:02:49.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:02:49.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:02:49.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:02:49.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:02:49.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reactivate/0 2024-04-25T08:02:49.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:02:49.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e142733d-38be-4e8f-8902-66e4bd59de9e 2024-04-25T08:02:49.871 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 e142733d-38be-4e8f-8902-66e4bd59de9e 2024-04-25T08:02:49.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e142733d-38be-4e8f-8902-66e4bd59de9e' 2024-04-25T08:02:49.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:02:49.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApDipmalrhNBAAzCdcL86OJwhDFdHueL+0nQ== 2024-04-25T08:02:49.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApDipmalrhNBAAzCdcL86OJwhDFdHueL+0nQ=="}' 2024-04-25T08:02:49.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e142733d-38be-4e8f-8902-66e4bd59de9e -i td/osd-reactivate/0/new.json 2024-04-25T08:02:50.206 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:02:50.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reactivate/0/new.json 2024-04-25T08:02:50.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=ee6a59fa-be7f-4597-87c1-bd567473cca1 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.202171/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApDipmalrhNBAAzCdcL86OJwhDFdHueL+0nQ== --osd-uuid e142733d-38be-4e8f-8902-66e4bd59de9e 2024-04-25T08:02:50.259 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:50.258+0000 7fbeff4067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:50.283 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:50.282+0000 7fbeff4067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:50.284 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:50.282+0000 7fbeff4067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:50.285 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:50.282+0000 7fbeff4067c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T08:02:50.285 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:50.282+0000 7fbeff4067c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T08:02:50.285 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:50.282+0000 7fbeff4067c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T08:02:50.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:50.322+0000 7fbeff4067c0 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2024-04-25T08:02:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reactivate/0/keyring 2024-04-25T08:02:56.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:02:56.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:02:56.891 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:02:56.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:02:57.473 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:02:57.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:02:57.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:02:57.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=ee6a59fa-be7f-4597-87c1-bd567473cca1 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.202171/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:02:57.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:02:57.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:02:57.500 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:57.498+0000 7fd9cdf6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:57.523 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:57.522+0000 7fd9cdf6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:57.524 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:57.522+0000 7fd9cdf6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:57.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:02:57.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:02:57.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:02:57.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:02:57.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:02:57.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:57.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:02:57.890 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:02:57.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:57.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:02:58.099 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:02:58.098+0000 7fd9cdf6e7c0 -1 Falling back to public interface 2024-04-25T08:02:58.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:59.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:59.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:59.386 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:02:59.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:02:59.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:59.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:02:59.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:00.798 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:00.798+0000 7fd9cdf6e7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:03:00.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:00.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:00.819 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:03:00.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:03:00.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:00.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:01.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:02.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:02.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:02.542 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:03:02.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:03:02.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:02.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:03.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:04.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:04.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:04.025 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:03:04.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:03:04.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:04.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:04.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:05.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:05.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:05.572 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:03:05.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:03:05.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:05.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:06.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:07.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:07.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:07.042 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:03:07.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:03:07.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:07.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:07.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:08.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:08.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:08.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:03:08.467 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:03:08.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:08.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:09.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:10.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:10.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:10.034 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:03:10.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:03:10.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:10.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:10.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:11.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:11.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:11.459 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:03:11.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:03:11.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:11.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:12.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:13.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:13.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:13.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:03:13.024 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:03:13.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:13.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:13.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:14.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:14.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:14.503 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:03:14.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:03:14.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:14.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:15.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:16.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:16.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:16.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:03:16.638 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:03:16.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:16.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:17.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:18.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:18.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:18.998 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:03:18.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:03:18.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:18.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:19.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:20.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:20.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:20.831 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:03:20.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:03:20.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:20.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:22.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:23.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:23.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:23.125 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:03:23.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:03:23.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:23.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:24.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:25.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:25.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:25.134 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:03:25.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:03:25.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:25.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:26.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:27.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:27.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:27.365 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:03:27.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:03:27.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:27.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:27.881 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2649136733,v1:127.0.0.1:6803/2649136733] [v2:127.0.0.1:6804/2649136733,v1:127.0.0.1:6805/2649136733] exists,up e142733d-38be-4e8f-8902-66e4bd59de9e 2024-04-25T08:03:27.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:03:27.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:03:27.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:03:27.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2024-04-25T08:03:27.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:03:27.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:03:27.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:03:27.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:03:27.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:03:28.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:03:28.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2024-04-25T08:03:28.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-reactivate 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-reactivate/0 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=ee6a59fa-be7f-4597-87c1-bd567473cca1 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-04-25T08:03:28.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202171 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.202171/$cluster-$name.asok' 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.202171/$cluster-$name.asok' 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-04-25T08:03:28.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:03:28.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:03:28.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:03:28.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:03:28.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:03:28.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:03:28.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-reactivate/0 2024-04-25T08:03:28.200 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:03:28.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T08:03:28.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=ee6a59fa-be7f-4597-87c1-bd567473cca1 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.202171/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:03:28.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-reactivate/0/whoami 2024-04-25T08:03:28.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T08:03:28.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:03:28.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:03:28.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:03:28.233 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:28.230+0000 7fcf150407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:28.256 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:28.254+0000 7fcf150407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:28.257 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:28.254+0000 7fcf150407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:28.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T08:03:28.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:03:28.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:03:28.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:03:28.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:03:28.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:28.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:03:28.641 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:03:28.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:28.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:28.831 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:28.830+0000 7fcf150407c0 -1 Falling back to public interface 2024-04-25T08:03:29.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:30.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:30.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:30.156 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:03:30.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:03:30.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:30.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:30.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:31.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:31.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:31.567 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:03:31.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:03:31.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:31.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:31.948 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:31.945+0000 7fcf150407c0 -1 osd.0 6 log_to_monitors true 2024-04-25T08:03:31.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:32.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:32.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:32.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:03:32.994 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:03:32.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:32.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:33.408 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/1688402582,v1:127.0.0.1:6803/1688402582] [v2:127.0.0.1:6804/1688402582,v1:127.0.0.1:6805/1688402582] exists,up e142733d-38be-4e8f-8902-66e4bd59de9e 2024-04-25T08:03:33.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:03:33.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:03:33.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:03:33.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2024-04-25T08:03:33.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T08:03:33.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:03:33.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T08:03:33.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:03:33.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:03:33.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:03:33.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:03:33.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:03:33.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:03:33.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:03:33.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:03:33.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:03:33.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:03:33.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:03:33.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:03:33.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:03:33.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:03:33.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:03:33.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:03:33.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:03:33.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:03:33.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T08:03:33.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:03:33.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:33.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202171 2024-04-25T08:03:33.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202171 2024-04-25T08:03:33.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:03:33.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:03:33.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:03:33.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-reactivate 0 2024-04-25T08:03:33.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T08:03:33.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:03:33.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T08:03:33.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:03:33.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:03:33.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:03:33.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:03:33.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:03:33.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:03:33.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:03:33.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:03:33.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:03:33.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:03:33.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:03:33.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:03:33.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:03:33.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:03:33.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:03:33.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:03:33.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:03:33.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:03:33.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T08:03:33.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:03:33.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:33.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.202171 2024-04-25T08:03:33.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.202171 2024-04-25T08:03:33.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:03:33.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:03:33.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:03:33.540 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:03:33.541 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:03:33.590 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2024-04-25T08:03:33.591 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2024-04-25T08:03:33.643 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-prio 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2024-04-25T08:03:33.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2024-04-25T08:03:33.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2024-04-25T08:03:33.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2024-04-25T08:03:33.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2024-04-25T08:03:33.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2024-04-25T08:03:33.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none ' 2024-04-25T08:03:33.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2024-04-25T08:03:33.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-04-25T08:03:33.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export objects=200 2024-04-25T08:03:33.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: objects=200 2024-04-25T08:03:33.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: export poolprefix=test 2024-04-25T08:03:33.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: poolprefix=test 2024-04-25T08:03:33.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: export FORCE_PRIO=255 2024-04-25T08:03:33.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: FORCE_PRIO=255 2024-04-25T08:03:33.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: export NORMAL_PRIO=190 2024-04-25T08:03:33.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: NORMAL_PRIO=190 2024-04-25T08:03:33.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: set 2024-04-25T08:03:33.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:03:33.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: local 'funcs=TEST_recovery_pool_priority 2024-04-25T08:03:33.653 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_recovery_priority' 2024-04-25T08:03:33.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-04-25T08:03:33.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-04-25T08:03:33.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-04-25T08:03:33.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-04-25T08:03:33.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:03:33.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:03:33.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:03:33.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:03:33.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:03:33.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:03:33.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:03:33.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:03:33.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:03:33.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:03:33.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:03:33.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:03:33.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:03:33.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:03:33.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:03:33.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:03:33.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:03:33.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:03:33.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:03:33.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:03:33.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:03:33.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:03:33.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:03:33.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:33.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:03:33.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.204781 2024-04-25T08:03:33.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:03:33.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:03:33.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-04-25T08:03:33.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:03:33.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:33.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:03:33.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.204781 2024-04-25T08:03:33.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:03:33.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:03:33.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:03:33.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:03:33.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-04-25T08:03:33.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_pool_priority td/osd-recovery-prio 2024-04-25T08:03:33.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:326: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2024-04-25T08:03:33.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: local pools=3 2024-04-25T08:03:33.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: local OSDS=2 2024-04-25T08:03:33.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:330: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2024-04-25T08:03:33.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-04-25T08:03:33.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:03:33.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:03:33.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:03:33.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-04-25T08:03:33.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-04-25T08:03:34.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:03:34.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:03:34.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:03:34.138 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:03:34.138 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:34.139 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:03:34.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:03:34.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:03:34.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:03:34.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:03:34.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:03:34.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:03:34.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:03:34.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:03:34.401 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:03:34.401 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:03:34.401 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:03:34.402 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:03:34.402 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:34.402 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:03:34.402 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-mon.a.asok 2024-04-25T08:03:34.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:03:34.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.204781/ceph-mon.a.asok config get fsid 2024-04-25T08:03:34.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:03:34.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:03:34.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:03:34.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:03:34.518 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:03:34.518 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:03:34.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:03:34.518 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:03:34.518 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:03:34.518 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:34.518 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:03:34.519 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-mon.a.asok 2024-04-25T08:03:34.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:03:34.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.204781/ceph-mon.a.asok config get mon_host 2024-04-25T08:03:34.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2024-04-25T08:03:34.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-04-25T08:03:34.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:03:34.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:03:34.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:03:34.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-04-25T08:03:34.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:03:35.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:03:35.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:03:35.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:03:35.537 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:03:35.537 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:35.537 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:03:35.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:03:35.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:03:35.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:03:35.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:332: TEST_recovery_pool_priority: export CEPH_ARGS 2024-04-25T08:03:35.569 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: expr 2 - 1 2024-04-25T08:03:35.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: seq 0 1 2024-04-25T08:03:35.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:03:35.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2024-04-25T08:03:35.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:03:35.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:03:35.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:03:35.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:03:35.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-04-25T08:03:35.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:03:35.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ea7b190-61de-4512-beb2-f6572d55b450 2024-04-25T08:03:35.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4ea7b190-61de-4512-beb2-f6572d55b450' 2024-04-25T08:03:35.577 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 4ea7b190-61de-4512-beb2-f6572d55b450 2024-04-25T08:03:35.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:03:35.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXDipmcWR8IxAAP9GmWCfdcWsPO4/QgGo4Ng== 2024-04-25T08:03:35.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXDipmcWR8IxAAP9GmWCfdcWsPO4/QgGo4Ng=="}' 2024-04-25T08:03:35.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ea7b190-61de-4512-beb2-f6572d55b450 -i td/osd-recovery-prio/0/new.json 2024-04-25T08:03:35.893 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:03:35.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-04-25T08:03:35.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXDipmcWR8IxAAP9GmWCfdcWsPO4/QgGo4Ng== --osd-uuid 4ea7b190-61de-4512-beb2-f6572d55b450 2024-04-25T08:03:35.946 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:35.945+0000 7f169c73a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:35.962 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:35.961+0000 7f169c73a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:35.967 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:35.961+0000 7f169c73a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:35.967 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:35.961+0000 7f169c73a7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:03:35.967 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:35.961+0000 7f169c73a7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:03:35.967 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:35.961+0000 7f169c73a7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:03:36.002 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:36.001+0000 7f169c73a7c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-04-25T08:03:42.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-04-25T08:03:42.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:03:42.473 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:03:42.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:03:42.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:03:43.012 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:03:43.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:03:43.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:03:43.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:03:43.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:03:43.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:03:43.046 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:43.045+0000 7f1cd3fe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:43.048 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:43.045+0000 7f1cd3fe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:43.049+0000 7f1cd3fe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:43.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:03:43.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:03:43.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:03:43.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:43.639 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:43.637+0000 7f1cd3fe47c0 -1 Falling back to public interface 2024-04-25T08:03:43.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:44.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:44.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:44.891 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:03:44.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:03:44.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:44.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:45.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:46.313 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:46.309+0000 7f1cd3fe47c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:03:46.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:46.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:46.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:03:46.387 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:03:46.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:46.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:46.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:47.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:47.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:47.880 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:03:47.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:03:47.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:47.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:48.307 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/305513354,v1:127.0.0.1:6803/305513354] [v2:127.0.0.1:6804/305513354,v1:127.0.0.1:6805/305513354] exists,up 4ea7b190-61de-4512-beb2-f6572d55b450 2024-04-25T08:03:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:03:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:03:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:03:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:03:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2024-04-25T08:03:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:03:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:03:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:03:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:03:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-04-25T08:03:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:48.312 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:03:48.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:03:48.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:03:48.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-04-25T08:03:48.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:03:48.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ad0cb669-bb01-484a-8785-2a93bc7787ee 2024-04-25T08:03:48.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ad0cb669-bb01-484a-8785-2a93bc7787ee' 2024-04-25T08:03:48.315 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 ad0cb669-bb01-484a-8785-2a93bc7787ee 2024-04-25T08:03:48.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:03:48.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBkDipm/GQgFBAAlPEtlUNFqo9WnfU49OmZHQ== 2024-04-25T08:03:48.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBkDipm/GQgFBAAlPEtlUNFqo9WnfU49OmZHQ=="}' 2024-04-25T08:03:48.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ad0cb669-bb01-484a-8785-2a93bc7787ee -i td/osd-recovery-prio/1/new.json 2024-04-25T08:03:48.890 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:03:48.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-04-25T08:03:48.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkDipm/GQgFBAAlPEtlUNFqo9WnfU49OmZHQ== --osd-uuid ad0cb669-bb01-484a-8785-2a93bc7787ee 2024-04-25T08:03:48.944 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:48.941+0000 7f57cecd27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:48.968 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:48.965+0000 7f57cecd27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:48.970 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:48.969+0000 7f57cecd27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:48.970 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:48.969+0000 7f57cecd27c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:03:48.970 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:48.969+0000 7f57cecd27c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:03:48.970 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:48.969+0000 7f57cecd27c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:03:49.008 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:49.005+0000 7f57cecd27c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-04-25T08:03:55.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-04-25T08:03:55.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:03:55.828 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T08:03:55.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:03:55.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:03:56.387 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:03:56.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:03:56.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:03:56.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:03:56.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:03:56.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:03:56.422 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:56.421+0000 7f6c8d89f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:56.441 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:56.437+0000 7f6c8d89f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:56.442 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:56.441+0000 7f6c8d89f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:56.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:03:56.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:03:56.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:03:56.845 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:03:56.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:03:56.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:03:56.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:56.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:03:56.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:56.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:03:57.019 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:57.017+0000 7f6c8d89f7c0 -1 Falling back to public interface 2024-04-25T08:03:57.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:58.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:58.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:58.322 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:03:58.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:03:58.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:58.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:03:58.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:59.657 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:03:59.653+0000 7f6c8d89f7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:03:59.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:59.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:59.763 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:03:59.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:03:59.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:59.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:00.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:01.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:01.271 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:04:01.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:04:01.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:01.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:01.695 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1320130799,v1:127.0.0.1:6811/1320130799] [v2:127.0.0.1:6812/1320130799,v1:127.0.0.1:6813/1320130799] exists,up ad0cb669-bb01-484a-8785-2a93bc7787ee 2024-04-25T08:04:01.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:04:01.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:04:01.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:04:01.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: seq 1 3 2024-04-25T08:04:01.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:04:01.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test1 1 1 2024-04-25T08:04:01.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T08:04:02.416 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test1' already exists 2024-04-25T08:04:02.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:04:03.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-04-25T08:04:04.249 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-04-25T08:04:04.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:04:04.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test2 1 1 2024-04-25T08:04:04.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-04-25T08:04:05.053 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test2' already exists 2024-04-25T08:04:05.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:04:06.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-04-25T08:04:07.054 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 2 size to 2 2024-04-25T08:04:07.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:04:07.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test3 1 1 2024-04-25T08:04:07.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-04-25T08:04:07.631 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test3' already exists 2024-04-25T08:04:07.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:04:08.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2024-04-25T08:04:09.456 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 3 size to 2 2024-04-25T08:04:09.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:344: TEST_recovery_pool_priority: sleep 5 2024-04-25T08:04:14.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:346: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T08:04:14.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:04:14.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:04:14.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:04:14.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:04:14.482 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:04:14.482 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:04:14.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:04:14.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:04:14.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:04:14.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:04:14.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:04:14.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:04:14.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:04:14.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:04:14.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:04:15.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:04:15.038 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-04-25T08:04:15.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:04:15.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:04:15.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:04:15.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836489 2024-04-25T08:04:15.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836489 2024-04-25T08:04:15.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489' 2024-04-25T08:04:15.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:04:15.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:04:15.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672966 2024-04-25T08:04:15.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672966 2024-04-25T08:04:15.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2024-04-25T08:04:15.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:04:15.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836489 2024-04-25T08:04:15.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:04:15.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:04:15.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836489 2024-04-25T08:04:15.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:04:15.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836489 2024-04-25T08:04:15.377 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836489 2024-04-25T08:04:15.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-04-25T08:04:15.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:04:15.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836489 2024-04-25T08:04:15.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:04:16.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:04:16.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:04:17.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836489 -lt 21474836489 2024-04-25T08:04:17.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:04:17.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:04:17.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672966 2024-04-25T08:04:17.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:04:17.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672966 2024-04-25T08:04:17.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:04:17.234 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672966 2024-04-25T08:04:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672966 2024-04-25T08:04:17.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-04-25T08:04:17.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:04:17.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672966 -lt 42949672966 2024-04-25T08:04:17.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:04:17.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:17.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:18.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T08:04:18.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:18.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:18.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:18.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:18.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:18.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:18.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:18.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T08:04:18.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:18.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:18.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:19.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T08:04:19.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:04:19.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:04:19.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:348: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:04:19.691 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:04:19.691 INFO:tasks.workunit.client.0.smithi102.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:04:10.320985+0000 0'0 28:20 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:04:07.461612+0000 0'0 2024-04-25T08:04:07.461612+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:35:30.425717+0000 0 0 2024-04-25T08:04:19.691 INFO:tasks.workunit.client.0.smithi102.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:04:07.812799+0000 0'0 28:29 [0,1] 0 [0,1] 0 0'0 2024-04-25T08:04:04.788866+0000 0'0 2024-04-25T08:04:04.788866+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:40:18.297947+0000 0 0 2024-04-25T08:04:19.691 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:04:04.812451+0000 0'0 28:41 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:04:02.168538+0000 0'0 2024-04-25T08:04:02.168538+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:13:19.557549+0000 0 0 2024-04-25T08:04:19.691 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:04:19.691 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:04:19.691 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:04:19.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local PG1 2024-04-25T08:04:19.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local POOLNUM1 2024-04-25T08:04:19.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:354: TEST_recovery_pool_priority: local pool1 2024-04-25T08:04:19.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local chk_osd1_1 2024-04-25T08:04:19.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local chk_osd1_2 2024-04-25T08:04:19.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local PG2 2024-04-25T08:04:19.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local POOLNUM2 2024-04-25T08:04:19.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:360: TEST_recovery_pool_priority: local pool2 2024-04-25T08:04:19.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: local chk_osd2_1 2024-04-25T08:04:19.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:362: TEST_recovery_pool_priority: local chk_osd2_2 2024-04-25T08:04:19.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: seq 1 3 2024-04-25T08:04:19.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:04:19.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2024-04-25T08:04:19.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-04-25T08:04:20.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:04:20.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=1 2024-04-25T08:04:20.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:04:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=0 2024-04-25T08:04:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z '' ']' 2024-04-25T08:04:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: PG1=1.0 2024-04-25T08:04:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: POOLNUM1=1 2024-04-25T08:04:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: pool1=test1 2024-04-25T08:04:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:374: TEST_recovery_pool_priority: chk_osd1_1=1 2024-04-25T08:04:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: chk_osd1_2=0 2024-04-25T08:04:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:04:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2024-04-25T08:04:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-04-25T08:04:20.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:04:20.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=0 2024-04-25T08:04:20.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:04:20.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=1 2024-04-25T08:04:20.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2024-04-25T08:04:20.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2024-04-25T08:04:20.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: PG2=2.0 2024-04-25T08:04:20.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: POOLNUM2=2 2024-04-25T08:04:20.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: pool2=test2 2024-04-25T08:04:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:381: TEST_recovery_pool_priority: chk_osd2_1=0 2024-04-25T08:04:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:382: TEST_recovery_pool_priority: chk_osd2_2=1 2024-04-25T08:04:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: break 2024-04-25T08:04:20.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:386: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2024-04-25T08:04:20.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:388: TEST_recovery_pool_priority: '[' test2 = '' ']' 2024-04-25T08:04:20.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: seq 1 3 2024-04-25T08:04:20.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:04:20.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2024-04-25T08:04:20.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:04:20.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2024-04-25T08:04:20.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:04:20.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2024-04-25T08:04:20.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:398: TEST_recovery_pool_priority: delete_pool test3 2024-04-25T08:04:20.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-04-25T08:04:20.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-04-25T08:04:21.110 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test3' does not exist 2024-04-25T08:04:21.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool1_extra_prio=1 2024-04-25T08:04:21.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:403: TEST_recovery_pool_priority: pool2_extra_prio=2 2024-04-25T08:04:21.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: expr 190 + 1 2024-04-25T08:04:21.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: pool1_prio=191 2024-04-25T08:04:21.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: expr 190 + 2 2024-04-25T08:04:21.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: pool2_prio=192 2024-04-25T08:04:21.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-04-25T08:04:22.113 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 1 2024-04-25T08:04:22.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2024-04-25T08:04:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 recovery_priority to 1 2024-04-25T08:04:23.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:409: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2024-04-25T08:04:24.123 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 2 size to 1 2024-04-25T08:04:24.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2024-04-25T08:04:25.019 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 2 recovery_priority to 2 2024-04-25T08:04:25.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T08:04:25.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:04:25.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:04:25.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:04:25.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:04:25.044 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:04:25.044 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:04:25.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:04:25.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:04:25.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:04:25.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:04:25.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:04:25.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:04:25.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:04:25.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:04:25.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:04:25.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:04:25.572 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-04-25T08:04:25.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:04:25.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:04:25.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:04:25.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836493 2024-04-25T08:04:25.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836493 2024-04-25T08:04:25.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493' 2024-04-25T08:04:25.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:04:25.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:04:25.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672970 2024-04-25T08:04:25.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672970 2024-04-25T08:04:25.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493 1-42949672970' 2024-04-25T08:04:25.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:04:25.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836493 2024-04-25T08:04:25.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:04:25.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:04:25.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836493 2024-04-25T08:04:25.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:04:25.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836493 2024-04-25T08:04:25.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-04-25T08:04:25.901 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836493 2024-04-25T08:04:25.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:04:26.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836493 -lt 21474836493 2024-04-25T08:04:26.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:04:26.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672970 2024-04-25T08:04:26.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:04:26.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:04:26.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672970 2024-04-25T08:04:26.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:04:26.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672970 2024-04-25T08:04:26.310 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672970 2024-04-25T08:04:26.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-04-25T08:04:26.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:04:26.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672970 -lt 42949672970 2024-04-25T08:04:26.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:04:26.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:26.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:27.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T08:04:27.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:04:27.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:04:27.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:04:27.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:04:27.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:04:27.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:04:27.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:04:27.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:04:27.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:04:27.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:04:27.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:04:28.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T08:04:28.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:04:28.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:04:28.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:413: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-04-25T08:04:28.312 INFO:tasks.workunit.client.0.smithi102.stderr:10+0 records in 2024-04-25T08:04:28.312 INFO:tasks.workunit.client.0.smithi102.stderr:10+0 records out 2024-04-25T08:04:28.312 INFO:tasks.workunit.client.0.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0388163 s, 270 MB/s 2024-04-25T08:04:28.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: p=1 2024-04-25T08:04:28.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-04-25T08:04:28.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-04-25T08:04:28.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:28.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-04-25T08:04:28.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:28.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-04-25T08:04:29.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:29.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-04-25T08:04:29.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:29.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-04-25T08:04:30.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:30.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-04-25T08:04:30.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:30.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-04-25T08:04:31.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:31.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-04-25T08:04:31.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:31.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-04-25T08:04:32.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:32.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-04-25T08:04:32.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:32.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-04-25T08:04:33.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:33.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-04-25T08:04:33.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:33.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-04-25T08:04:33.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:33.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-04-25T08:04:34.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:34.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-04-25T08:04:34.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:34.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-04-25T08:04:35.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:35.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-04-25T08:04:35.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:35.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-04-25T08:04:35.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:35.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-04-25T08:04:36.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:36.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-04-25T08:04:36.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:36.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-04-25T08:04:37.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:37.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-04-25T08:04:37.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:37.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-04-25T08:04:38.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:38.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-04-25T08:04:38.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:38.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-04-25T08:04:39.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:39.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-04-25T08:04:39.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:39.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-04-25T08:04:39.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:39.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-04-25T08:04:40.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:40.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-04-25T08:04:40.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:40.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-04-25T08:04:41.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:41.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-04-25T08:04:41.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:41.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-04-25T08:04:42.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:42.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-04-25T08:04:42.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:42.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-04-25T08:04:43.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:43.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-04-25T08:04:43.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:43.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-04-25T08:04:43.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:43.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-04-25T08:04:44.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:44.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-04-25T08:04:44.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:44.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-04-25T08:04:45.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:45.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-04-25T08:04:45.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:45.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-04-25T08:04:46.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:46.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-04-25T08:04:46.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:46.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-04-25T08:04:47.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:47.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-04-25T08:04:47.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:47.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-04-25T08:04:48.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:48.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-04-25T08:04:48.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:48.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-04-25T08:04:48.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:48.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-04-25T08:04:49.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-04-25T08:04:49.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:49.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-04-25T08:04:50.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:50.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-04-25T08:04:50.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:50.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-04-25T08:04:51.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:51.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-04-25T08:04:51.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:51.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-04-25T08:04:52.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:52.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-04-25T08:04:52.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:52.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-04-25T08:04:53.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:53.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-04-25T08:04:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-04-25T08:04:53.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:53.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-04-25T08:04:54.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:54.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-04-25T08:04:54.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:54.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-04-25T08:04:55.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:55.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-04-25T08:04:55.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:55.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-04-25T08:04:56.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:56.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-04-25T08:04:56.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:56.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-04-25T08:04:56.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:56.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-04-25T08:04:57.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:57.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-04-25T08:04:57.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:57.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-04-25T08:04:58.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:58.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-04-25T08:04:58.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:58.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-04-25T08:04:59.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:59.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-04-25T08:04:59.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:59.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-04-25T08:04:59.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:04:59.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-04-25T08:05:00.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:00.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-04-25T08:05:00.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:00.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-04-25T08:05:01.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:01.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-04-25T08:05:02.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:02.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-04-25T08:05:02.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:02.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-04-25T08:05:03.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:03.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-04-25T08:05:03.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:03.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-04-25T08:05:03.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:03.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-04-25T08:05:04.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:04.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-04-25T08:05:04.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:04.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-04-25T08:05:05.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:05.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-04-25T08:05:05.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:05.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-04-25T08:05:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:06.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-04-25T08:05:06.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:06.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-04-25T08:05:07.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:07.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-04-25T08:05:07.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:07.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-04-25T08:05:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-04-25T08:05:08.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:08.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-04-25T08:05:09.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:09.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-04-25T08:05:09.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:09.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-04-25T08:05:09.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:09.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-04-25T08:05:10.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:10.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-04-25T08:05:10.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:10.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-04-25T08:05:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-04-25T08:05:11.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:11.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-04-25T08:05:12.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:12.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-04-25T08:05:12.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:12.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-04-25T08:05:13.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:13.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-04-25T08:05:13.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:13.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-04-25T08:05:13.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:13.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-04-25T08:05:14.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:14.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-04-25T08:05:14.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:14.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-04-25T08:05:15.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:15.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-04-25T08:05:15.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:15.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-04-25T08:05:16.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:16.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-04-25T08:05:16.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:16.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-04-25T08:05:17.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:17.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-04-25T08:05:17.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:17.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-04-25T08:05:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:18.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-04-25T08:05:18.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:18.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-04-25T08:05:18.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:18.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-04-25T08:05:19.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:19.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-04-25T08:05:19.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:19.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-04-25T08:05:20.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:20.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-04-25T08:05:20.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:20.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-04-25T08:05:21.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:21.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-04-25T08:05:21.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:21.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-04-25T08:05:22.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:22.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-04-25T08:05:22.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:22.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-04-25T08:05:23.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:23.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-04-25T08:05:23.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:23.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-04-25T08:05:23.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:23.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-04-25T08:05:24.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:24.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-04-25T08:05:24.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:24.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-04-25T08:05:25.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:25.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-04-25T08:05:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:25.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-04-25T08:05:26.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:26.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-04-25T08:05:26.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:26.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-04-25T08:05:27.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:27.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-04-25T08:05:27.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:27.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-04-25T08:05:27.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:27.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-04-25T08:05:28.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:28.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-04-25T08:05:28.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:28.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-04-25T08:05:29.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:29.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-04-25T08:05:29.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:29.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-04-25T08:05:30.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:30.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-04-25T08:05:30.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:30.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-04-25T08:05:31.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:31.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-04-25T08:05:31.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:31.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-04-25T08:05:31.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:31.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-04-25T08:05:32.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:32.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-04-25T08:05:33.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:33.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-04-25T08:05:33.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:33.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-04-25T08:05:33.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:33.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-04-25T08:05:34.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:34.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-04-25T08:05:34.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:34.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-04-25T08:05:35.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:35.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-04-25T08:05:35.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:35.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-04-25T08:05:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-04-25T08:05:36.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:36.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-04-25T08:05:37.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:37.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-04-25T08:05:37.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:37.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-04-25T08:05:38.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:38.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-04-25T08:05:38.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:38.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-04-25T08:05:39.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:39.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-04-25T08:05:39.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:39.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-04-25T08:05:40.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:40.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-04-25T08:05:40.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:40.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-04-25T08:05:41.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:41.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-04-25T08:05:41.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:41.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-04-25T08:05:41.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:41.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-04-25T08:05:42.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:42.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-04-25T08:05:42.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:42.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-04-25T08:05:43.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:43.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-04-25T08:05:43.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:43.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-04-25T08:05:44.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:44.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-04-25T08:05:44.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:44.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-04-25T08:05:45.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:45.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-04-25T08:05:45.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:45.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-04-25T08:05:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-04-25T08:05:46.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:46.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-04-25T08:05:46.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:46.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-04-25T08:05:47.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:47.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-04-25T08:05:47.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:47.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-04-25T08:05:48.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:48.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-04-25T08:05:48.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:48.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-04-25T08:05:49.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:49.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-04-25T08:05:49.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:49.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-04-25T08:05:50.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:50.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-04-25T08:05:50.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:50.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-04-25T08:05:51.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:51.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-04-25T08:05:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-04-25T08:05:51.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:51.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-04-25T08:05:52.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:52.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-04-25T08:05:52.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:52.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-04-25T08:05:53.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:53.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-04-25T08:05:53.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:53.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-04-25T08:05:54.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:54.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-04-25T08:05:54.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:54.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-04-25T08:05:55.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:55.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-04-25T08:05:55.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:55.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-04-25T08:05:56.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:56.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-04-25T08:05:56.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:56.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-04-25T08:05:56.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:56.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-04-25T08:05:57.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:57.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-04-25T08:05:57.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:57.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-04-25T08:05:58.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:58.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-04-25T08:05:58.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:58.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-04-25T08:05:59.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 1 + 1 2024-04-25T08:05:59.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=2 2024-04-25T08:05:59.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-04-25T08:05:59.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-04-25T08:05:59.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:59.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2024-04-25T08:05:59.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:05:59.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2024-04-25T08:06:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2024-04-25T08:06:00.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:00.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2024-04-25T08:06:01.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:01.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2024-04-25T08:06:01.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:01.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2024-04-25T08:06:02.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:02.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2024-04-25T08:06:02.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:02.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2024-04-25T08:06:03.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:03.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2024-04-25T08:06:03.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:03.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2024-04-25T08:06:03.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:03.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2024-04-25T08:06:04.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:04.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2024-04-25T08:06:04.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:04.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2024-04-25T08:06:05.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:05.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2024-04-25T08:06:05.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:05.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2024-04-25T08:06:06.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:06.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2024-04-25T08:06:06.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:06.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2024-04-25T08:06:07.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:07.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2024-04-25T08:06:07.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:07.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2024-04-25T08:06:08.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2024-04-25T08:06:08.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:08.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2024-04-25T08:06:09.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:09.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2024-04-25T08:06:09.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:09.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2024-04-25T08:06:09.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:09.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2024-04-25T08:06:10.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:10.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2024-04-25T08:06:10.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:10.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2024-04-25T08:06:11.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:11.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2024-04-25T08:06:11.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:11.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2024-04-25T08:06:12.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:12.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2024-04-25T08:06:12.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:12.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2024-04-25T08:06:13.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:13.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2024-04-25T08:06:13.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:13.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2024-04-25T08:06:14.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:14.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2024-04-25T08:06:14.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:14.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2024-04-25T08:06:15.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:15.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2024-04-25T08:06:15.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:15.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2024-04-25T08:06:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2024-04-25T08:06:16.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:16.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2024-04-25T08:06:16.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:16.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2024-04-25T08:06:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:17.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2024-04-25T08:06:17.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:17.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2024-04-25T08:06:18.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:18.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2024-04-25T08:06:18.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:18.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2024-04-25T08:06:19.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:19.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2024-04-25T08:06:19.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:19.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2024-04-25T08:06:20.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:20.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2024-04-25T08:06:20.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:20.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2024-04-25T08:06:20.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:20.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2024-04-25T08:06:21.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:21.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2024-04-25T08:06:21.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:21.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2024-04-25T08:06:22.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:22.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2024-04-25T08:06:22.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:22.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2024-04-25T08:06:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:23.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2024-04-25T08:06:23.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:23.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2024-04-25T08:06:24.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:24.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2024-04-25T08:06:24.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:24.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2024-04-25T08:06:25.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:25.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2024-04-25T08:06:25.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:25.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2024-04-25T08:06:25.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:25.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2024-04-25T08:06:26.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:26.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2024-04-25T08:06:26.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:26.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2024-04-25T08:06:27.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:27.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2024-04-25T08:06:27.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:27.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2024-04-25T08:06:28.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:28.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2024-04-25T08:06:28.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:28.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2024-04-25T08:06:29.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:29.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2024-04-25T08:06:29.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:29.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2024-04-25T08:06:29.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:29.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2024-04-25T08:06:30.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:30.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2024-04-25T08:06:30.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:30.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2024-04-25T08:06:31.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:31.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2024-04-25T08:06:31.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:31.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2024-04-25T08:06:32.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:32.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2024-04-25T08:06:32.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:32.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2024-04-25T08:06:33.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:33.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2024-04-25T08:06:33.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:33.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2024-04-25T08:06:33.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:33.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2024-04-25T08:06:34.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:34.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2024-04-25T08:06:35.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:35.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2024-04-25T08:06:35.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:35.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2024-04-25T08:06:35.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:35.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2024-04-25T08:06:36.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:36.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2024-04-25T08:06:36.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:36.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2024-04-25T08:06:37.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:37.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2024-04-25T08:06:37.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:37.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2024-04-25T08:06:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2024-04-25T08:06:38.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:38.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2024-04-25T08:06:39.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:39.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2024-04-25T08:06:39.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:39.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2024-04-25T08:06:39.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:39.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2024-04-25T08:06:40.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:40.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2024-04-25T08:06:40.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:40.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2024-04-25T08:06:41.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:41.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2024-04-25T08:06:41.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:41.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2024-04-25T08:06:42.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:42.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2024-04-25T08:06:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:42.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2024-04-25T08:06:43.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:43.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2024-04-25T08:06:43.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:43.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2024-04-25T08:06:43.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:43.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2024-04-25T08:06:44.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:44.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2024-04-25T08:06:44.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:44.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2024-04-25T08:06:45.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:45.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2024-04-25T08:06:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:45.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2024-04-25T08:06:46.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:46.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2024-04-25T08:06:46.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:46.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2024-04-25T08:06:47.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:47.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2024-04-25T08:06:47.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:47.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2024-04-25T08:06:47.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:47.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2024-04-25T08:06:48.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:48.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2024-04-25T08:06:48.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:48.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2024-04-25T08:06:49.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:49.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2024-04-25T08:06:49.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:49.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2024-04-25T08:06:50.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:50.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2024-04-25T08:06:50.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:50.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2024-04-25T08:06:51.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:51.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2024-04-25T08:06:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2024-04-25T08:06:52.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:52.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2024-04-25T08:06:52.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:52.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2024-04-25T08:06:53.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:53.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2024-04-25T08:06:53.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:53.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2024-04-25T08:06:53.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:53.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2024-04-25T08:06:54.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:54.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2024-04-25T08:06:54.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:54.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2024-04-25T08:06:55.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:55.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2024-04-25T08:06:55.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:55.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2024-04-25T08:06:56.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:56.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2024-04-25T08:06:56.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:56.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2024-04-25T08:06:57.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:57.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2024-04-25T08:06:57.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:57.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2024-04-25T08:06:57.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:57.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2024-04-25T08:06:58.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:58.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2024-04-25T08:06:58.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:58.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2024-04-25T08:06:59.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:59.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2024-04-25T08:06:59.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:06:59.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2024-04-25T08:07:00.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:00.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2024-04-25T08:07:00.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:00.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2024-04-25T08:07:01.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:01.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2024-04-25T08:07:01.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:01.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2024-04-25T08:07:01.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:01.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2024-04-25T08:07:02.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:02.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2024-04-25T08:07:02.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:02.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2024-04-25T08:07:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2024-04-25T08:07:03.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:03.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2024-04-25T08:07:03.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:03.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2024-04-25T08:07:04.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:04.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2024-04-25T08:07:04.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:04.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2024-04-25T08:07:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:05.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2024-04-25T08:07:05.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:05.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2024-04-25T08:07:06.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:06.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2024-04-25T08:07:06.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:06.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2024-04-25T08:07:07.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:07.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2024-04-25T08:07:07.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:07.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2024-04-25T08:07:08.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:08.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2024-04-25T08:07:08.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:08.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2024-04-25T08:07:09.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:09.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2024-04-25T08:07:09.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:09.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2024-04-25T08:07:10.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:10.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2024-04-25T08:07:10.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:10.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2024-04-25T08:07:11.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:11.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2024-04-25T08:07:11.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:11.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2024-04-25T08:07:11.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:11.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2024-04-25T08:07:12.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:12.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2024-04-25T08:07:12.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:12.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2024-04-25T08:07:13.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:13.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2024-04-25T08:07:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2024-04-25T08:07:14.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:14.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2024-04-25T08:07:14.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:14.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2024-04-25T08:07:15.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:15.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2024-04-25T08:07:15.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:15.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2024-04-25T08:07:15.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:15.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2024-04-25T08:07:16.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:16.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2024-04-25T08:07:16.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:16.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2024-04-25T08:07:17.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:17.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2024-04-25T08:07:17.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:17.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2024-04-25T08:07:18.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:18.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2024-04-25T08:07:18.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:18.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2024-04-25T08:07:19.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:19.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2024-04-25T08:07:19.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:19.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2024-04-25T08:07:20.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:20.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2024-04-25T08:07:20.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:20.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2024-04-25T08:07:21.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:21.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2024-04-25T08:07:21.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:21.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2024-04-25T08:07:21.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:21.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2024-04-25T08:07:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:22.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2024-04-25T08:07:22.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:22.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2024-04-25T08:07:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:23.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2024-04-25T08:07:23.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:23.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2024-04-25T08:07:24.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:24.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2024-04-25T08:07:24.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:24.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2024-04-25T08:07:25.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:25.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2024-04-25T08:07:25.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:25.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2024-04-25T08:07:26.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:26.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2024-04-25T08:07:26.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:26.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2024-04-25T08:07:26.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:26.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2024-04-25T08:07:27.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:27.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2024-04-25T08:07:27.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:27.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2024-04-25T08:07:28.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:28.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2024-04-25T08:07:28.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:28.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2024-04-25T08:07:29.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:29.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2024-04-25T08:07:29.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:07:29.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2024-04-25T08:07:30.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 2 + 1 2024-04-25T08:07:30.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=3 2024-04-25T08:07:30.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2024-04-25T08:07:30.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-04-25T08:07:30.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-04-25T08:07:30.065 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-04-25T08:07:30.066 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-04-25T08:07:30.066 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-04-25T08:07:30.066 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:07:30.066 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:07:30.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:07:30.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:07:30.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:07:30.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: local otherosd=null 2024-04-25T08:07:30.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:07:31.395 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:07:31.396 INFO:tasks.workunit.client.0.smithi102.stdout:2.0 195 0 0 0 0 2042626048 0 0 584 0 584 active+clean 2024-04-25T08:04:24.627635+0000 39'584 39:641 [0] 0 [0] 0 0'0 2024-04-25T08:04:04.788866+0000 0'0 2024-04-25T08:04:04.788866+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:34:23.071199+0000 0 0 2024-04-25T08:07:31.396 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:04:22.718135+0000 39'600 39:666 [1] 1 [1] 1 0'0 2024-04-25T08:04:02.168538+0000 0'0 2024-04-25T08:04:02.168538+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:17:44.070073+0000 0 0 2024-04-25T08:07:31.396 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:07:31.396 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:07:31.396 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:07:31.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ERRORS=0 2024-04-25T08:07:31.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-04-25T08:07:32.187 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-04-25T08:07:32.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-04-25T08:07:33.573 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 2 size to 2 2024-04-25T08:07:33.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:07:34.103 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:07:34.103 INFO:tasks.workunit.client.0.smithi102.stdout:2.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:04:24.627635+0000 39'600 42:662 [0] 0 [0] 0 0'0 2024-04-25T08:04:04.788866+0000 0'0 2024-04-25T08:04:04.788866+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:34:23.071199+0000 0 0 2024-04-25T08:07:34.103 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:04:22.718135+0000 39'600 39:666 [1] 1 [1] 1 0'0 2024-04-25T08:04:02.168538+0000 0'0 2024-04-25T08:04:02.168538+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:17:44.070073+0000 0 0 2024-04-25T08:07:34.103 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:07:34.103 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:07:34.104 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:07:34.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:436: TEST_recovery_pool_priority: set -o pipefail 2024-04-25T08:07:34.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: count=0 2024-04-25T08:07:34.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T08:07:34.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T08:07:34.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T08:07:34.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T08:07:34.657 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:07:34.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 0 == 2 2024-04-25T08:07:34.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-04-25T08:07:36.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 0 -eq 10 2024-04-25T08:07:36.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 0 + 1 2024-04-25T08:07:36.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=1 2024-04-25T08:07:36.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T08:07:36.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T08:07:36.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T08:07:36.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T08:07:37.165 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:07:37.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 0 == 2 2024-04-25T08:07:37.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-04-25T08:07:39.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 1 -eq 10 2024-04-25T08:07:39.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 1 + 1 2024-04-25T08:07:39.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=2 2024-04-25T08:07:39.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T08:07:39.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T08:07:39.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T08:07:39.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T08:07:39.838 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:07:39.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 2 == 2 2024-04-25T08:07:39.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: break 2024-04-25T08:07:39.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: set +o pipefail 2024-04-25T08:07:39.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:07:40.237 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:07:40.237 INFO:tasks.workunit.client.0.smithi102.stdout:2.0 200 0 199 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-04-25T08:07:36.516790+0000 39'600 47:674 [0,1] 0 [0] 0 0'0 2024-04-25T08:04:04.788866+0000 0'0 2024-04-25T08:04:04.788866+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:07:40.237 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 200 0 195 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-04-25T08:07:33.539253+0000 39'600 47:693 [1,0] 1 [1] 1 0'0 2024-04-25T08:04:02.168538+0000 0'0 2024-04-25T08:04:02.168538+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:07:40.238 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:07:40.238 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:07:40.238 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:07:40.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.1 2024-04-25T08:07:40.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:07:40.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:07:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:07:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:07:40.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.1.asok 2024-04-25T08:07:40.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2024-04-25T08:07:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:07:40.376 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 2024-04-25T08:07:40.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.1 2024-04-25T08:07:40.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:07:40.377 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:07:40.377 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:07:40.377 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:07:40.377 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:07:40.377 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:07:40.377 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:07:40.377 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:07:40.377 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:07:40.377 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 191, 2024-04-25T08:07:40.377 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:07:40.377 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "2.0", 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 192, 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:07:40.378 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:07:40.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: get_asok_path osd.0 2024-04-25T08:07:40.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T08:07:40.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T08:07:40.380 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:07:40.380 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:40.380 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:07:40.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.0.asok 2024-04-25T08:07:40.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: CEPH_ARGS= 2024-04-25T08:07:40.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.0.asok dump_recovery_reservations 2024-04-25T08:07:40.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:460: TEST_recovery_pool_priority: echo osd.0 2024-04-25T08:07:40.493 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 2024-04-25T08:07:40.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:07:40.494 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:07:40.494 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:07:40.494 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:07:40.494 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:07:40.494 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:07:40.494 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:07:40.494 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:07:40.494 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "2.0", 2024-04-25T08:07:40.494 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 192, 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 191, 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:07:40.495 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:07:40.496 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:07:40.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:07:40.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:07:40.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-04-25T08:07:40.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: ITEM=1.0 2024-04-25T08:07:40.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:465: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-04-25T08:07:40.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:07:40.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:07:40.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: PRIO=191 2024-04-25T08:07:40.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:471: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-04-25T08:07:40.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:07:40.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-04-25T08:07:40.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-04-25T08:07:40.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: ITEM=1.0 2024-04-25T08:07:40.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:480: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-04-25T08:07:40.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:07:40.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-04-25T08:07:40.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: PRIO=191 2024-04-25T08:07:40.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:486: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-04-25T08:07:40.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:07:40.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:07:40.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-04-25T08:07:40.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: ITEM=2.0 2024-04-25T08:07:40.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:495: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-04-25T08:07:40.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:07:40.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:07:40.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: PRIO=192 2024-04-25T08:07:40.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:501: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-04-25T08:07:40.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-04-25T08:07:40.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:07:40.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-04-25T08:07:40.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: ITEM=2.0 2024-04-25T08:07:40.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:510: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-04-25T08:07:40.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:07:40.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-04-25T08:07:40.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: PRIO=192 2024-04-25T08:07:40.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:516: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-04-25T08:07:40.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:523: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T08:07:40.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:07:40.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:07:40.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:07:40.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:07:40.777 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:07:40.777 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:07:40.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:07:40.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:07:40.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:07:40.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:07:40.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:07:40.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:07:40.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:07:40.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:07:40.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:07:41.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:07:41.463 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-04-25T08:07:41.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:07:41.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:07:41.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:07:41.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836534 2024-04-25T08:07:41.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836534 2024-04-25T08:07:41.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534' 2024-04-25T08:07:41.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:07:41.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:07:42.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673012 2024-04-25T08:07:42.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673012 2024-04-25T08:07:42.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836534 1-42949673012' 2024-04-25T08:07:42.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:07:42.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836534 2024-04-25T08:07:42.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:07:42.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:07:42.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836534 2024-04-25T08:07:42.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:07:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836534 2024-04-25T08:07:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-04-25T08:07:42.121 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836534 2024-04-25T08:07:42.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:07:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836534 -lt 21474836534 2024-04-25T08:07:42.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:07:42.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673012 2024-04-25T08:07:42.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:07:42.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:07:42.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673012 2024-04-25T08:07:42.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:07:42.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673012 2024-04-25T08:07:42.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2024-04-25T08:07:42.918 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673012 2024-04-25T08:07:42.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:07:43.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673011 -lt 42949673012 2024-04-25T08:07:43.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:07:44.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:07:44.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:07:45.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673012 -lt 42949673012 2024-04-25T08:07:45.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:07:45.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:07:45.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:07:46.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T08:07:46.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:07:46.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:07:46.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:07:46.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:07:46.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:07:46.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:07:46.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:07:46.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:07:46.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:07:46.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:07:46.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:07:47.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:07:47.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:07:47.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:07:47.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:07:47.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:07:47.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:07:47.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:07:47.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:07:47.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:07:47.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:07:47.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:07:47.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:07:47.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:07:47.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:07:48.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:07:48.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:07:48.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:07:48.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:07:48.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:07:48.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:07:48.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:07:48.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:07:48.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:07:48.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:07:48.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:07:48.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:07:48.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:07:49.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31447679 2024-04-25T08:07:49.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31447679 '!=' null 2024-04-25T08:07:49.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:07:49.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:07:49.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:07:49.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:07:49.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:07:49.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:07:49.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:07:49.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:07:49.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:07:49.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:07:49.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:07:50.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:07:50.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:07:50.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:07:50.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:07:50.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:07:50.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:07:50.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:07:50.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:07:50.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:07:50.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:07:50.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:07:50.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:07:50.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:07:51.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26205975 2024-04-25T08:07:51.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26205975 '!=' null 2024-04-25T08:07:51.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:07:51.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:07:51.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:07:51.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:07:51.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:07:51.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:07:51.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:07:51.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:07:51.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:07:51.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:07:51.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:07:51.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:07:51.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:07:51.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:07:51.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:07:52.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:07:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:07:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:07:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:07:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:07:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:07:52.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:07:52.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:07:52.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:07:53.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31447658 2024-04-25T08:07:53.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31447658 '!=' null 2024-04-25T08:07:53.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:07:53.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:07:53.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:07:53.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:07:53.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:07:53.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:07:53.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:07:53.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:07:53.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:07:53.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:07:53.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:07:53.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:07:53.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:07:53.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:07:53.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:07:54.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:07:54.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:07:54.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:07:54.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:07:54.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:07:54.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:07:54.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:07:54.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:07:54.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:07:55.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459196 2024-04-25T08:07:55.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459196 '!=' null 2024-04-25T08:07:55.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:07:55.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:07:55.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:07:55.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:07:55.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:07:55.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:07:55.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:07:55.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:07:55.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:07:55.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:07:55.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:07:55.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:07:55.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:07:55.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:07:55.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:07:56.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:07:56.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:07:56.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:07:56.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:07:56.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:07:56.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:07:56.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:07:56.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:07:56.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:07:57.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585703 2024-04-25T08:07:57.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585703 '!=' null 2024-04-25T08:07:57.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:07:57.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:07:57.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:07:57.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:07:57.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:07:57.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:07:57.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:07:57.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:07:57.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:07:57.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:07:57.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:07:57.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:07:57.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:07:57.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:07:57.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:07:58.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:07:58.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:07:58.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:07:58.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:07:58.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:07:58.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:07:58.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:07:58.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:07:58.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:07:59.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34068463 2024-04-25T08:07:59.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34068463 '!=' null 2024-04-25T08:07:59.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:07:59.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:07:59.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:07:59.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:07:59.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:07:59.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:07:59.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:07:59.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:07:59.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:07:59.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:07:59.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:07:59.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:07:59.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:07:59.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:07:59.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:00.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:00.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:00.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:00.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:00.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:00.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:00.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:00.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:00.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:01.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965358 2024-04-25T08:08:01.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965358 '!=' null 2024-04-25T08:08:01.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:01.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:01.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:01.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:01.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:01.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:01.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:01.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:01.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:01.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:01.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:01.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:01.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:01.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:01.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:02.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:02.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:02.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:03.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206858 2024-04-25T08:08:03.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206858 '!=' null 2024-04-25T08:08:03.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:03.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:03.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:03.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:03.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:03.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:03.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:03.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:03.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:03.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:03.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:03.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:03.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:03.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:03.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:04.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:04.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:04.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:04.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:04.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:04.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:04.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:04.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:04.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:05.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206554 2024-04-25T08:08:05.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206554 '!=' null 2024-04-25T08:08:05.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:05.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:05.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:05.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:05.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:05.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:05.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:05.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:05.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:05.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:05.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:05.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:05.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:05.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:05.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:06.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:06.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:06.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:06.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:06.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:06.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:07.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965232 2024-04-25T08:08:07.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965232 '!=' null 2024-04-25T08:08:07.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:07.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:07.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:07.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:07.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:07.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:07.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:07.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:07.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:07.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:07.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:07.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:07.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:07.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:07.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:08.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:08.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:08.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:08.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:08.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:08.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31447944 2024-04-25T08:08:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31447944 '!=' null 2024-04-25T08:08:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:09.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:09.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:09.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:09.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:09.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:09.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:09.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:09.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:09.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:09.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:09.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:09.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:10.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:10.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:10.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:10.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:10.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:10.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:10.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:10.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:10.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:11.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965302 2024-04-25T08:08:11.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965302 '!=' null 2024-04-25T08:08:11.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:11.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:11.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:11.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:11.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:11.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:11.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:11.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:11.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:11.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:11.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:11.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:11.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:11.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:11.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:12.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:12.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:12.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:12.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:12.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:12.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:12.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:12.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:12.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:13.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206749 2024-04-25T08:08:13.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206749 '!=' null 2024-04-25T08:08:13.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:13.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:13.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:13.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:13.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:13.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:13.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:13.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:13.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:13.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:13.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:13.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:13.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:13.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:13.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:14.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:14.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:15.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207243 2024-04-25T08:08:15.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207243 '!=' null 2024-04-25T08:08:15.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:15.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:15.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:15.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:15.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:15.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:15.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:15.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:15.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:15.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:15.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:15.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:15.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:15.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:15.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:16.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:16.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:16.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:16.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:16.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:16.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:16.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:16.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:16.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:17.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965864 2024-04-25T08:08:17.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965864 '!=' null 2024-04-25T08:08:17.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:17.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:17.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:17.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:17.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:17.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:17.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:17.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:17.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:17.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:17.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:17.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:17.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:17.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:17.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:18.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:18.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:18.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:18.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:18.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:18.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:18.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:18.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:18.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:19.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31448166 2024-04-25T08:08:19.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31448166 '!=' null 2024-04-25T08:08:19.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:19.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:19.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:19.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:19.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:19.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:19.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:19.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:19.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:19.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:19.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:19.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:19.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:19.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:19.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:20.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:20.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:20.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:20.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:20.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:20.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:20.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:20.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:20.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:21.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965357 2024-04-25T08:08:21.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965357 '!=' null 2024-04-25T08:08:21.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:21.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:21.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:21.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:21.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:21.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:21.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:21.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:21.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:21.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:21.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:21.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:21.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:21.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:21.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:22.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:22.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:22.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:22.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:22.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:22.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:22.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:22.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:22.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:23.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206395 2024-04-25T08:08:23.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206395 '!=' null 2024-04-25T08:08:23.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:23.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:23.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:23.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:23.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:23.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:23.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:23.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:23.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:23.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:23.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:23.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:23.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:23.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:23.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:24.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:24.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:24.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:24.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:24.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:24.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:24.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:24.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:24.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:25.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206060 2024-04-25T08:08:25.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206060 '!=' null 2024-04-25T08:08:25.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:25.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:25.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:25.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:25.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:25.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:25.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:25.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:25.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:25.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:25.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:26.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:26.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:26.039 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:26.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:26.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:26.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:26.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:26.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:26.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:26.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:26.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:26.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:26.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:27.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20964570 2024-04-25T08:08:27.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20964570 '!=' null 2024-04-25T08:08:27.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:27.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:27.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:27.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:27.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:27.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:27.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:27.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:27.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:27.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:27.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:27.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:27.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:27.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:27.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:28.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:28.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:28.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:28.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:28.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:28.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:28.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:28.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:28.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:29.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31446911 2024-04-25T08:08:29.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31446911 '!=' null 2024-04-25T08:08:29.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:29.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:29.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:29.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:29.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:29.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:29.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:29.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:29.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:29.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:29.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:29.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:29.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:29.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:29.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:30.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:30.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:30.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:30.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:30.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:30.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:30.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:30.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:30.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:31.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965076 2024-04-25T08:08:31.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965076 '!=' null 2024-04-25T08:08:31.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:31.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:31.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:31.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:31.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:31.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:31.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:31.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:31.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:31.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:31.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:31.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:31.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:31.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:31.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:32.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:32.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:32.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:32.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:32.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:32.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:32.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:32.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:32.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207073 2024-04-25T08:08:32.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207073 '!=' null 2024-04-25T08:08:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:32.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:32.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:32.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:32.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:32.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:32.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:32.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:32.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:32.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:33.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:33.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:33.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:33.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:33.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:33.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:33.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:33.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:33.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:33.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:33.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:33.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:33.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:34.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207073 2024-04-25T08:08:34.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207073 '!=' null 2024-04-25T08:08:34.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:34.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:34.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:34.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:34.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:34.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:34.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:34.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:34.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:34.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:34.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:35.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:35.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:35.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:35.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:35.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:35.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:36.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207389 2024-04-25T08:08:36.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207389 '!=' null 2024-04-25T08:08:36.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:36.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:36.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:36.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:36.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:36.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:36.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:36.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:36.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:36.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:36.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:36.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:36.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:36.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:36.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:37.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:37.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:37.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:37.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:37.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:37.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:37.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:37.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:37.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:38.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20966142 2024-04-25T08:08:38.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20966142 '!=' null 2024-04-25T08:08:38.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:38.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:38.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:38.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:38.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:38.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:38.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:38.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:38.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:38.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:38.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:38.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:38.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:38.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:39.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:39.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:39.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:39.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:39.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:39.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:39.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:39.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:39.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=32322827 2024-04-25T08:08:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 32322827 '!=' null 2024-04-25T08:08:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:40.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:40.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:40.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:40.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:40.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:40.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:40.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:40.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:40.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:40.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:40.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:40.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:40.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:41.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:41.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:41.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:41.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:42.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839670 2024-04-25T08:08:42.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839670 '!=' null 2024-04-25T08:08:42.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:42.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:42.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:42.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:42.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:42.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:42.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:42.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:42.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:42.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:42.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:42.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:42.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:42.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:42.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:43.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:43.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:43.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:43.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:43.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:43.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:43.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:43.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:43.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:44.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207535 2024-04-25T08:08:44.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207535 '!=' null 2024-04-25T08:08:44.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:44.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:44.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:44.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:44.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:44.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:44.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:44.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:44.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:44.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:44.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:44.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:44.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:44.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:44.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:45.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:45.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:45.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:45.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:45.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:45.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:45.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:45.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:45.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:46.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080310 2024-04-25T08:08:46.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080310 '!=' null 2024-04-25T08:08:46.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:46.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:46.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:46.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:46.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:46.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:46.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:46.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:46.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:46.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:46.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:46.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:46.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:46.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:46.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:47.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:47.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:47.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:47.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:47.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:47.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:47.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:47.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:47.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:48.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838842 2024-04-25T08:08:48.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838842 '!=' null 2024-04-25T08:08:48.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:48.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:48.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:48.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:48.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:48.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:48.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:48.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:48.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:48.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:48.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:48.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:48.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:48.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:48.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:49.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:49.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:49.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:49.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:49.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:49.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:49.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:49.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:49.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:50.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33194674 2024-04-25T08:08:50.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33194674 '!=' null 2024-04-25T08:08:50.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:50.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:50.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:50.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:50.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:50.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:50.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:50.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:50.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:50.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:50.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:50.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:50.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:50.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:50.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:51.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:51.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:51.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:51.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:51.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:51.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:51.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:51.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:51.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:52.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838073 2024-04-25T08:08:52.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838073 '!=' null 2024-04-25T08:08:52.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:52.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:52.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:52.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:52.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:52.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:52.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:52.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:52.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:52.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:52.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:52.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:52.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:52.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:52.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:53.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:53.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:53.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:53.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:53.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:53.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:53.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:53.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:53.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:53.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27079628 2024-04-25T08:08:53.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27079628 '!=' null 2024-04-25T08:08:53.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:53.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:53.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:54.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:54.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:54.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:54.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:54.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:54.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:54.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:54.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:54.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:54.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:54.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:54.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:55.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:55.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:55.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:55.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:55.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:55.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:55.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:55.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:55.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:56.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27953840 2024-04-25T08:08:56.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27953840 '!=' null 2024-04-25T08:08:56.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:56.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:56.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:56.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:56.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:56.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:56.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:56.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:56.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:56.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:56.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:56.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:56.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:56.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:56.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:57.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:57.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:57.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:57.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:57.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:57.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:57.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:57.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:57.048 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:57.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839315 2024-04-25T08:08:57.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839315 '!=' null 2024-04-25T08:08:57.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:57.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:57.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:57.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:57.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:57.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:57.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:57.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:57.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:57.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:57.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:58.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:08:58.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:58.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:58.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:58.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:08:58.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:08:58.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:08:58.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:08:58.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:08:58.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:08:58.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:08:58.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:08:58.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:08:59.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=30575017 2024-04-25T08:08:59.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 30575017 '!=' null 2024-04-25T08:08:59.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:08:59.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:08:59.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:08:59.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:08:59.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:59.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:59.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:59.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:59.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:59.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:59.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:00.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:00.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:00.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:00.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:00.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:00.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:00.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:00.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:00.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:00.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:00.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:00.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:00.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:01.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19218598 2024-04-25T08:09:01.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19218598 '!=' null 2024-04-25T08:09:01.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:01.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:01.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:01.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:01.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:01.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:01.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:01.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:01.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:01.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:01.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:02.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:02.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:02.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:02.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:02.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:02.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:02.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:02.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:02.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:02.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:02.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:02.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:02.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:03.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333944 2024-04-25T08:09:03.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333944 '!=' null 2024-04-25T08:09:03.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:03.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:03.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:03.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:03.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:03.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:03.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:03.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:03.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:03.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:03.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:03.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:03.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:03.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:03.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:04.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:04.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:04.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:04.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:04.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:04.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:04.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:04.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:04.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:05.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207284 2024-04-25T08:09:05.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207284 '!=' null 2024-04-25T08:09:05.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:05.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:05.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:05.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:05.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:05.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:05.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:05.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:05.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:05.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:05.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:05.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:05.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:05.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:05.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:06.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:06.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:06.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:06.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:06.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:06.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:06.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:06.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:06.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:06.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965035 2024-04-25T08:09:06.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965035 '!=' null 2024-04-25T08:09:06.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:06.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:06.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:07.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:07.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:07.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:07.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:07.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:07.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:07.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:07.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:07.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:07.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:07.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:07.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:08.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:08.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:09.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31447560 2024-04-25T08:09:09.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31447560 '!=' null 2024-04-25T08:09:09.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:09.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:09.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:09.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:09.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:09.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:09.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:09.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:09.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:09.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:09.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:09.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:09.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:09.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:09.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:10.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:10.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:11.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712143 2024-04-25T08:09:11.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712143 '!=' null 2024-04-25T08:09:11.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:11.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:11.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:11.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:11.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:11.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:11.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:11.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:11.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:11.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:11.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:11.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:11.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:11.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:11.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:12.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:12.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:12.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:12.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:12.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:12.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:12.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:12.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:12.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:13.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27953325 2024-04-25T08:09:13.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27953325 '!=' null 2024-04-25T08:09:13.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:13.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:13.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:13.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:13.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:13.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:13.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:13.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:13.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:13.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:13.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:13.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:13.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:13.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:13.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:14.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:14.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:14.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:14.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:14.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:14.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:14.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:14.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:14.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:14.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27079965 2024-04-25T08:09:14.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27079965 '!=' null 2024-04-25T08:09:14.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:14.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:14.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:14.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:14.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:14.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:14.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:14.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:14.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:14.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:14.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:15.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:15.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:15.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:15.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:15.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:15.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:15.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:16.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27079965 2024-04-25T08:09:16.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27079965 '!=' null 2024-04-25T08:09:16.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:16.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:16.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:16.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:16.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:16.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:16.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:16.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:16.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:16.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:16.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:17.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:17.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:17.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:17.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:17.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:17.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:17.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:17.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:17.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:17.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:17.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:17.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:18.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31447972 2024-04-25T08:09:18.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31447972 '!=' null 2024-04-25T08:09:18.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:18.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:18.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:18.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:18.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:18.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:18.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:18.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:18.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:18.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:18.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:19.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:19.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:19.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:19.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:19.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:19.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:19.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:19.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:19.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:19.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:19.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:19.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:19.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:20.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31447972 2024-04-25T08:09:20.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31447972 '!=' null 2024-04-25T08:09:20.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:20.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:20.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:20.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:20.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:20.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:20.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:20.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:20.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:20.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:20.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:21.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:21.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:21.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:21.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:21.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:21.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:21.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:21.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:21.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:21.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:21.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:21.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:21.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:22.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206062 2024-04-25T08:09:22.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206062 '!=' null 2024-04-25T08:09:22.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:22.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:22.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:22.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:22.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:22.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:22.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:22.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:22.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:22.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:22.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:23.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:23.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:23.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:23.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:23.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:23.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:23.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:23.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:23.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:23.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:23.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:23.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:23.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:24.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206062 2024-04-25T08:09:24.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206062 '!=' null 2024-04-25T08:09:24.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:24.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:24.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:24.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:24.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:24.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:24.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:24.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:24.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:24.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:24.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:24.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:24.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:24.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:24.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:25.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:25.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:25.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:25.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:25.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:25.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:25.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:25.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:25.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:26.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26205802 2024-04-25T08:09:26.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26205802 '!=' null 2024-04-25T08:09:26.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:26.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:26.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:26.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:26.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:26.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:26.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:26.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:26.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:26.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:26.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:26.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:26.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:26.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:26.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:27.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:27.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:27.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:27.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:27.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:27.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:27.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:27.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:27.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:28.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20964414 2024-04-25T08:09:28.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20964414 '!=' null 2024-04-25T08:09:28.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:28.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:28.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:28.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:28.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:28.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:28.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:28.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:28.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:28.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:28.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:28.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:28.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:28.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:28.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:29.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:29.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:29.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:29.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:29.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:29.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:29.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:29.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:29.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:29.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=32320381 2024-04-25T08:09:29.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 32320381 '!=' null 2024-04-25T08:09:29.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:29.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:29.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:30.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:30.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:30.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:30.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:30.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:30.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:30.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:30.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:30.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:30.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:30.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:30.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:31.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:31.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:31.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:31.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:31.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:31.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:31.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:31.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:31.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:31.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838345 2024-04-25T08:09:31.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838345 '!=' null 2024-04-25T08:09:31.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:31.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:31.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:31.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:31.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:31.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:31.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:31.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:31.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:31.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:31.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:32.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:32.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:32.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:32.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:32.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:32.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:32.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:32.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:32.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:32.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:32.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:32.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:32.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:33.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27079821 2024-04-25T08:09:33.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27079821 '!=' null 2024-04-25T08:09:33.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:33.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:33.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:34.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:34.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:34.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:34.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:34.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:34.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:34.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:34.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:34.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:34.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:34.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:34.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:35.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:35.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:35.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:35.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:35.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:35.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:35.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:35.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:35.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:35.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27953476 2024-04-25T08:09:35.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27953476 '!=' null 2024-04-25T08:09:35.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:35.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:35.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:35.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:35.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:35.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:35.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:35.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:35.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:35.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:35.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:36.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:36.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:36.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:36.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:36.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:36.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:36.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:36.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:36.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:36.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:36.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:36.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:36.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:37.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712813 2024-04-25T08:09:37.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712813 '!=' null 2024-04-25T08:09:37.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:37.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:37.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:37.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:37.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:37.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:37.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:37.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:37.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:37.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:37.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:38.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:38.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:38.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:38.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:38.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:38.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:38.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:38.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:38.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:38.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:38.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:38.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:38.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:39.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33196100 2024-04-25T08:09:39.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33196100 '!=' null 2024-04-25T08:09:39.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:39.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:39.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:39.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:39.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:39.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:39.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:39.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:39.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:39.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:39.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:40.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:40.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:40.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:40.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:40.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:40.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:40.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:40.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:40.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:40.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:40.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:40.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:40.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:41.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839815 2024-04-25T08:09:41.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839815 '!=' null 2024-04-25T08:09:41.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:41.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:41.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:41.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:41.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:41.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:41.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:41.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:41.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:41.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:41.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:41.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:41.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:41.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:41.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:42.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:42.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:42.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:42.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:42.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:42.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:42.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:42.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:42.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:43.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27081736 2024-04-25T08:09:43.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27081736 '!=' null 2024-04-25T08:09:43.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:43.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:43.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:43.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:43.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:43.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:43.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:43.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:43.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:43.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:43.287 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:43.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:43.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:43.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:43.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:44.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:44.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:44.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:44.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:44.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:44.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:44.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:44.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:44.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:45.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27082280 2024-04-25T08:09:45.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27082280 '!=' null 2024-04-25T08:09:45.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:45.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:45.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:45.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:45.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:45.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:45.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:45.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:45.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:45.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:45.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:45.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:45.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:45.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:45.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:46.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:46.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:46.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:46.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:46.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:46.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:46.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:46.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:46.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:47.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20966795 2024-04-25T08:09:47.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20966795 '!=' null 2024-04-25T08:09:47.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:47.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:47.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:47.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:47.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:47.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:47.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:47.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:47.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:47.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:47.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:47.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:47.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:47.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:47.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:48.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:48.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:48.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:48.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:48.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:48.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:48.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:48.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:48.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:49.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31449392 2024-04-25T08:09:49.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31449392 '!=' null 2024-04-25T08:09:49.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:49.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:49.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:49.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:49.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:49.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:49.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:49.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:49.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:49.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:49.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:49.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:49.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:49.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:49.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:50.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:50.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:50.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:50.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:50.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:50.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:50.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:50.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:50.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20966178 2024-04-25T08:09:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20966178 '!=' null 2024-04-25T08:09:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:51.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:51.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:51.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:51.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:51.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:51.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:51.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:51.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:52.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:52.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:52.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:52.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:52.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:52.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:53.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207215 2024-04-25T08:09:53.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207215 '!=' null 2024-04-25T08:09:53.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:53.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:53.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:53.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:53.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:53.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:53.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:53.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:53.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:53.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:53.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:54.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:54.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:54.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:54.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:55.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:55.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:55.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:55.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:55.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:55.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:55.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:55.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:55.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:55.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206677 2024-04-25T08:09:55.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206677 '!=' null 2024-04-25T08:09:55.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:55.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:55.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:55.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:55.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:55.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:55.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:55.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:55.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:55.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:55.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:56.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:56.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:56.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:56.183 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:56.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:56.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:56.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:56.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:56.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:56.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:56.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:56.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:56.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:57.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965439 2024-04-25T08:09:57.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965439 '!=' null 2024-04-25T08:09:57.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:57.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:57.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:57.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:57.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:57.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:57.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:57.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:57.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:57.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:57.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:57.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:57.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:57.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:09:57.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:58.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:09:58.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:09:58.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:09:58.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:09:58.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:09:58.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:09:58.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:09:58.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:09:58.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:09:59.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31448343 2024-04-25T08:09:59.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31448343 '!=' null 2024-04-25T08:09:59.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:09:59.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:09:59.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:09:59.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:09:59.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:09:59.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:09:59.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:09:59.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:09:59.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:09:59.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:09:59.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:09:59.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:09:59.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:09:59.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:09:59.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:00.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:10:00.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:10:00.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:00.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:00.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:00.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:00.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:00.340 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:00.340 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:01.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965470 2024-04-25T08:10:01.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965470 '!=' null 2024-04-25T08:10:01.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:01.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:01.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:01.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:01.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:01.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:01.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:01.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:01.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:01.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:01.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:01.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:10:01.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:01.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:01.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:02.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:10:02.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:10:02.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:02.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:02.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:02.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:02.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:02.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:02.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:03.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206902 2024-04-25T08:10:03.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206902 '!=' null 2024-04-25T08:10:03.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:03.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:03.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:03.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:03.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:03.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:03.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:03.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:03.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:03.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:03.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:03.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:10:03.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:03.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:03.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:04.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:10:04.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:10:04.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:04.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:04.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:04.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:04.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:04.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:04.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:04.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206877 2024-04-25T08:10:04.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206877 '!=' null 2024-04-25T08:10:04.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:04.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:04.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:05.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:05.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:05.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:05.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:05.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:05.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:05.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:05.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:05.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:10:05.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:05.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:05.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:06.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:10:06.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:10:06.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:06.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:06.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:06.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:06.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:06.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:06.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:06.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965478 2024-04-25T08:10:06.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965478 '!=' null 2024-04-25T08:10:06.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:06.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:06.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:06.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:06.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:06.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:06.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:06.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:06.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:06.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:06.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:07.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:10:07.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:07.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:07.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:08.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:10:08.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:10:08.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:08.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:08.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:08.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:08.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:08.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:08.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:08.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31448076 2024-04-25T08:10:08.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31448076 '!=' null 2024-04-25T08:10:08.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:08.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:08.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:08.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:08.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:08.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:08.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:08.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:08.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:08.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:08.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:10:09.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:09.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:09.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:09.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:10:09.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:10:09.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:09.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:09.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:09.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:09.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:09.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:09.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:10.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=31448076 2024-04-25T08:10:10.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 31448076 '!=' null 2024-04-25T08:10:10.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:10.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:10.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:10.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:10.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:10.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:10.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:10.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:10.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:10.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:10.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:11.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:10:11.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:11.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:11.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:11.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:10:11.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:10:11.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:11.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:11.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:11.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:11.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:11.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:11.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:12.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965336 2024-04-25T08:10:12.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965336 '!=' null 2024-04-25T08:10:12.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:12.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:12.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:12.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:12.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:12.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:12.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:12.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:12.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:12.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:12.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:13.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:10:13.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:13.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:13.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:14.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:10:14.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:10:14.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:14.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:14.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:14.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:14.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:14.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:14.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:14.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=32759803 2024-04-25T08:10:14.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 32759803 '!=' null 2024-04-25T08:10:14.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:14.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:14.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:14.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:14.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:14.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:14.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:14.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:14.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:14.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:14.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:15.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:10:15.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:15.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:15.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:16.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:10:16.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:10:16.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:16.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:16.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:16.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:16.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:16.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:16.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:16.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17035293 2024-04-25T08:10:16.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17035293 '!=' null 2024-04-25T08:10:16.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:16.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:16.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:16.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:16.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:16.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:16.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:16.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:16.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:16.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:16.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:17.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:10:17.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:17.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:17.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:18.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:10:18.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:10:18.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:18.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:18.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:18.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:18.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:18.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:18.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28875877 2024-04-25T08:10:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28875877 '!=' null 2024-04-25T08:10:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:18.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:18.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:18.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:18.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:18.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:18.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:18.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:18.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:18.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:18.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:19.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:10:19.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:19.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:19.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:19.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:10:19.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-04-25T08:10:19.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:10:19.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-04-25T08:10:19.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:19.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:19.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:19.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:19.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:19.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:19.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:19.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:19.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:19.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:20.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:10:20.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:20.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:20.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:20.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:10:20.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:10:20.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:20.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:20.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:20.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:20.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:20.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:20.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:21.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25149802 2024-04-25T08:10:21.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25149802 '!=' null 2024-04-25T08:10:21.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:21.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:21.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:21.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:21.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:21.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:21.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:21.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:21.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:21.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:21.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:21.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:10:21.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:21.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:21.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:22.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:10:22.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:10:22.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:22.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:22.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:22.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:22.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:22.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:22.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14414984 2024-04-25T08:10:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14414984 '!=' null 2024-04-25T08:10:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:23.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:23.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:23.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:23.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:23.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:23.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:23.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:23.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:23.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:10:23.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:23.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:23.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:23.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:10:23.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:10:23.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:10:23.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:10:23.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:10:23.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:10:23.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:10:24.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:10:24.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:10:24.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14414620 2024-04-25T08:10:24.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14414620 '!=' null 2024-04-25T08:10:24.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:10:24.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:10:24.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:10:24.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:10:24.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:10:24.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:10:24.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:10:24.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:10:24.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:10:24.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:10:24.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:10:25.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:10:25.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:10:25.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:10:25.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:10:25.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T08:10:25.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:10:25.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:10:25.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:525: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2024-04-25T08:10:25.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:529: TEST_recovery_pool_priority: echo TEST PASSED 2024-04-25T08:10:25.733 INFO:tasks.workunit.client.0.smithi102.stdout:TEST PASSED 2024-04-25T08:10:25.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:532: TEST_recovery_pool_priority: delete_pool test1 2024-04-25T08:10:25.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T08:10:25.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T08:10:26.266 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test1' does not exist 2024-04-25T08:10:26.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:533: TEST_recovery_pool_priority: delete_pool test2 2024-04-25T08:10:26.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-04-25T08:10:26.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-04-25T08:10:26.887 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test2' does not exist 2024-04-25T08:10:26.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:534: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2024-04-25T08:10:26.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:10:26.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:10:26.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:10:26.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:10:26.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:10:32.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:10:32.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:535: TEST_recovery_pool_priority: return 0 2024-04-25T08:10:32.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-04-25T08:10:32.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:10:32.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:10:32.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:10:32.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:10:32.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:10:32.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:10:32.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:10:32.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:10:32.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:10:32.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:10:32.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:10:32.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:10:32.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:10:32.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:10:32.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:10:32.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:10:32.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:10:32.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:10:32.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:10:32.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:10:32.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:10:32.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:10:32.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:10:32.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:10:32.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:10:32.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.204781 2024-04-25T08:10:32.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:10:32.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:10:32.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-04-25T08:10:32.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-04-25T08:10:32.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-04-25T08:10:32.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-04-25T08:10:32.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:10:32.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:10:32.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:10:32.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:10:32.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:10:32.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:10:32.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:10:32.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:10:32.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:10:32.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:10:32.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:10:32.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:10:32.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:10:32.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:10:32.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:10:32.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:10:32.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:10:32.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:10:32.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:10:32.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:10:32.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:10:32.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:10:32.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:10:32.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:10:32.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:10:32.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.204781 2024-04-25T08:10:32.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:10:32.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:10:32.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-04-25T08:10:32.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:10:32.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:10:32.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:10:32.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.204781 2024-04-25T08:10:32.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:10:32.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:10:32.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:10:32.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-04-25T08:10:32.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_priority td/osd-recovery-prio 2024-04-25T08:10:32.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local dir=td/osd-recovery-prio 2024-04-25T08:10:32.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:48: TEST_recovery_priority: local pools=10 2024-04-25T08:10:32.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: local OSDS=5 2024-04-25T08:10:32.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: local max_tries=10 2024-04-25T08:10:32.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:52: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2024-04-25T08:10:32.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-04-25T08:10:32.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:10:32.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:10:32.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:10:32.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-04-25T08:10:32.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-04-25T08:10:32.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:10:32.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:10:32.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:10:32.787 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:10:32.787 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:10:32.787 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:10:32.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:10:32.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:10:33.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:10:33.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:10:33.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:10:33.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:10:33.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:10:33.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:10:33.025 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:10:33.025 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:10:33.025 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:10:33.025 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:10:33.025 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:10:33.025 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:10:33.026 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-mon.a.asok 2024-04-25T08:10:33.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:10:33.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.204781/ceph-mon.a.asok config get fsid 2024-04-25T08:10:33.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:10:33.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:10:33.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:10:33.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:10:33.142 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:10:33.142 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:10:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:10:33.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:10:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:10:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:10:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:10:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-mon.a.asok 2024-04-25T08:10:33.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:10:33.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.204781/ceph-mon.a.asok config get mon_host 2024-04-25T08:10:33.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2024-04-25T08:10:33.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-04-25T08:10:33.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:10:33.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:10:33.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:10:33.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-04-25T08:10:33.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:10:34.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:10:34.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:10:34.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:10:34.189 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:10:34.189 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:10:34.189 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:10:34.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:10:34.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:10:34.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:10:34.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:54: TEST_recovery_priority: export CEPH_ARGS 2024-04-25T08:10:34.224 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: expr 5 - 1 2024-04-25T08:10:34.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: seq 0 4 2024-04-25T08:10:34.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:10:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2024-04-25T08:10:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:10:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:10:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:10:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:10:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-04-25T08:10:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:10:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:10:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:10:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:10:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:10:34.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-04-25T08:10:34.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:10:34.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=08d944a3-6290-4ecb-bb54-16629faddf8e 2024-04-25T08:10:34.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 08d944a3-6290-4ecb-bb54-16629faddf8e' 2024-04-25T08:10:34.232 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 08d944a3-6290-4ecb-bb54-16629faddf8e 2024-04-25T08:10:34.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:10:34.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6DypmUXjiDhAALDAOrYwMfirXoTFAXHC25Q== 2024-04-25T08:10:34.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6DypmUXjiDhAALDAOrYwMfirXoTFAXHC25Q=="}' 2024-04-25T08:10:34.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 08d944a3-6290-4ecb-bb54-16629faddf8e -i td/osd-recovery-prio/0/new.json 2024-04-25T08:10:34.567 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:10:34.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-04-25T08:10:34.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6DypmUXjiDhAALDAOrYwMfirXoTFAXHC25Q== --osd-uuid 08d944a3-6290-4ecb-bb54-16629faddf8e 2024-04-25T08:10:34.628 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:34.624+0000 7fc0fbb947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:34.644 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:34.640+0000 7fc0fbb947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:34.645 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:34.640+0000 7fc0fbb947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:34.645 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:34.640+0000 7fc0fbb947c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:10:34.645 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:34.640+0000 7fc0fbb947c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:10:34.645 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:34.640+0000 7fc0fbb947c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:10:34.684 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:34.680+0000 7fc0fbb947c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-04-25T08:10:41.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-04-25T08:10:41.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:10:41.065 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:10:41.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:10:41.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:10:41.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:10:41.629 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:10:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:10:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:10:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:10:41.657 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:41.652+0000 7f9b234427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:41.659 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:41.656+0000 7f9b234427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:41.660 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:41.656+0000 7f9b234427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:42.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:10:42.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:10:42.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:10:42.032 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:10:42.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:10:42.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:10:42.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:42.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:10:42.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:10:42.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:42.248 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:42.244+0000 7f9b234427c0 -1 Falling back to public interface 2024-04-25T08:10:42.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:43.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:43.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:43.470 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:10:43.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:10:43.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:43.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:10:43.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:44.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:44.820+0000 7f9b234427c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:10:44.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:44.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:44.883 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:10:44.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:10:44.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:44.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:10:45.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:46.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:46.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:46.408 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:10:46.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:10:46.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:46.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:10:46.839 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1080874133,v1:127.0.0.1:6803/1080874133] [v2:127.0.0.1:6804/1080874133,v1:127.0.0.1:6805/1080874133] exists,up 08d944a3-6290-4ecb-bb54-16629faddf8e 2024-04-25T08:10:46.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:10:46.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:10:46.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:10:46.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-04-25T08:10:46.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:10:46.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62b86227-9ed0-4912-b243-f56d8a0d1889 2024-04-25T08:10:46.846 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 62b86227-9ed0-4912-b243-f56d8a0d1889 2024-04-25T08:10:46.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 62b86227-9ed0-4912-b243-f56d8a0d1889' 2024-04-25T08:10:46.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:10:46.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAGECpm0oXSMxAAkAYhm+pl7yP1ky6uecwUjg== 2024-04-25T08:10:46.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAGECpm0oXSMxAAkAYhm+pl7yP1ky6uecwUjg=="}' 2024-04-25T08:10:46.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62b86227-9ed0-4912-b243-f56d8a0d1889 -i td/osd-recovery-prio/1/new.json 2024-04-25T08:10:47.357 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:10:47.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-04-25T08:10:47.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGECpm0oXSMxAAkAYhm+pl7yP1ky6uecwUjg== --osd-uuid 62b86227-9ed0-4912-b243-f56d8a0d1889 2024-04-25T08:10:47.411 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:47.408+0000 7f06b8d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:47.413 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:47.408+0000 7f06b8d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:47.414 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:47.412+0000 7f06b8d1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:47.414 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:47.412+0000 7f06b8d1f7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:10:47.414 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:47.412+0000 7f06b8d1f7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:10:47.414 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:47.412+0000 7f06b8d1f7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:10:47.482 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:47.476+0000 7f06b8d1f7c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-04-25T08:10:54.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-04-25T08:10:54.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:10:54.385 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T08:10:54.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:10:54.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:10:54.992 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:10:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:10:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:10:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:10:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:10:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:10:55.026 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:55.024+0000 7fd78ccdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:55.028 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:55.024+0000 7fd78ccdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:55.029 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:55.024+0000 7fd78ccdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:10:55.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:10:55.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:10:55.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:10:55.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:10:55.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:10:55.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:55.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:10:55.432 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:10:55.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:55.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:10:55.612 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:55.608+0000 7fd78ccdd7c0 -1 Falling back to public interface 2024-04-25T08:10:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:56.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:56.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:56.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:10:56.877 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:10:56.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:56.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:10:57.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:58.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:10:58.032+0000 7fd78ccdd7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:10:58.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:58.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:58.289 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:10:58.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:10:58.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:58.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:10:58.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:10:59.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:10:59.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:10:59.787 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:10:59.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:10:59.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:10:59.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:11:00.235 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3040329781,v1:127.0.0.1:6811/3040329781] [v2:127.0.0.1:6812/3040329781,v1:127.0.0.1:6813/3040329781] exists,up 62b86227-9ed0-4912-b243-f56d8a0d1889 2024-04-25T08:11:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:11:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:11:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:11:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:11:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2024-04-25T08:11:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:11:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:11:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:11:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:11:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/2 2024-04-25T08:11:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:11:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:11:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:11:00.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:11:00.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:11:00.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/2 2024-04-25T08:11:00.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:11:00.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c9e9cab9-86e2-42db-b361-bb58a03c2060 2024-04-25T08:11:00.243 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 c9e9cab9-86e2-42db-b361-bb58a03c2060 2024-04-25T08:11:00.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c9e9cab9-86e2-42db-b361-bb58a03c2060' 2024-04-25T08:11:00.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:11:00.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUECpmXqWFDxAAPOdaiQJxfLEL8KnPKZEsLg== 2024-04-25T08:11:00.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUECpmXqWFDxAAPOdaiQJxfLEL8KnPKZEsLg=="}' 2024-04-25T08:11:00.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c9e9cab9-86e2-42db-b361-bb58a03c2060 -i td/osd-recovery-prio/2/new.json 2024-04-25T08:11:00.781 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:11:00.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/2/new.json 2024-04-25T08:11:00.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUECpmXqWFDxAAPOdaiQJxfLEL8KnPKZEsLg== --osd-uuid c9e9cab9-86e2-42db-b361-bb58a03c2060 2024-04-25T08:11:00.836 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:00.832+0000 7fc0237a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:00.866 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:00.864+0000 7fc0237a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:00.868 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:00.864+0000 7fc0237a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:00.868 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:00.864+0000 7fc0237a47c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T08:11:00.868 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:00.864+0000 7fc0237a47c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T08:11:00.868 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:00.864+0000 7fc0237a47c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T08:11:00.984 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:00.980+0000 7fc0237a47c0 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2024-04-25T08:11:08.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2024-04-25T08:11:08.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:11:08.037 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T08:11:08.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:11:08.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:11:08.582 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T08:11:08.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:11:08.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:11:08.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:11:08.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:11:08.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:11:08.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:08.612+0000 7f1e3f0917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:08.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:08.616+0000 7f1e3f0917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:08.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:08.616+0000 7f1e3f0917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:09.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:11:09.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:11:09.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:11:09.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:11:09.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:11:09.048 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:11:09.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:09.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:11:09.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:11:09.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:09.179 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:09.176+0000 7f1e3f0917c0 -1 Falling back to public interface 2024-04-25T08:11:09.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:10.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:10.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:10.464 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:11:10.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:11:10.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:10.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:11:10.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:11.895 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:11.892+0000 7f1e3f0917c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:11:11.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:11.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:11.910 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:11:11.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:11:11.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:11.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:11:12.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:13.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:13.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:13.315 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:11:13.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:11:13.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:13.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:11:13.722 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/449594563,v1:127.0.0.1:6819/449594563] [v2:127.0.0.1:6820/449594563,v1:127.0.0.1:6821/449594563] exists,up c9e9cab9-86e2-42db-b361-bb58a03c2060 2024-04-25T08:11:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:11:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:11:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:11:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:11:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2024-04-25T08:11:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:11:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:11:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:11:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:11:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/3 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:11:13.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:11:13.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/3 2024-04-25T08:11:13.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:11:13.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=628e530d-201b-44d3-be04-4bdb8228f923 2024-04-25T08:11:13.728 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 628e530d-201b-44d3-be04-4bdb8228f923 2024-04-25T08:11:13.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 628e530d-201b-44d3-be04-4bdb8228f923' 2024-04-25T08:11:13.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:11:13.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhECpmW4bCLBAAsN5COGNmK7aP57IDxxR23Q== 2024-04-25T08:11:13.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhECpmW4bCLBAAsN5COGNmK7aP57IDxxR23Q=="}' 2024-04-25T08:11:13.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 628e530d-201b-44d3-be04-4bdb8228f923 -i td/osd-recovery-prio/3/new.json 2024-04-25T08:11:14.273 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:11:14.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/3/new.json 2024-04-25T08:11:14.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhECpmW4bCLBAAsN5COGNmK7aP57IDxxR23Q== --osd-uuid 628e530d-201b-44d3-be04-4bdb8228f923 2024-04-25T08:11:14.319 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:14.316+0000 7f29db4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:14.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:14.316+0000 7f29db4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:14.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:14.320+0000 7f29db4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:14.323 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:14.320+0000 7f29db4e27c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T08:11:14.323 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:14.320+0000 7f29db4e27c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T08:11:14.323 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:14.320+0000 7f29db4e27c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T08:11:14.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:14.404+0000 7f29db4e27c0 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2024-04-25T08:11:21.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2024-04-25T08:11:21.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:11:21.325 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T08:11:21.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:11:21.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:11:21.890 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T08:11:21.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:11:21.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:11:21.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:11:21.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:11:21.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:11:21.924 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:21.923+0000 7f8314acd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:21.942 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:21.939+0000 7f8314acd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:21.943 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:21.943+0000 7f8314acd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:22.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:11:22.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:11:22.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:11:22.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:11:22.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:11:22.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:22.311 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:11:22.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:11:22.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:22.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:11:22.496 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:22.495+0000 7f8314acd7c0 -1 Falling back to public interface 2024-04-25T08:11:22.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:23.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:23.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:23.732 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:11:23.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:11:23.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:23.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:11:24.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:25.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:25.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:25.142 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:11:25.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:11:25.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:25.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:11:25.233 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:25.231+0000 7f8314acd7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:11:25.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:26.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:26.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:26.595 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:11:26.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:11:26.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:26.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:11:27.006 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3461930747,v1:127.0.0.1:6827/3461930747] [v2:127.0.0.1:6828/3461930747,v1:127.0.0.1:6829/3461930747] exists,up 628e530d-201b-44d3-be04-4bdb8228f923 2024-04-25T08:11:27.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:11:27.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/4 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2024-04-25T08:11:27.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2024-04-25T08:11:27.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:11:27.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:11:27.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:11:27.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:11:27.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:11:27.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:11:27.010 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:11:27.010 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:11:27.010 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:11:27.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:11:27.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/4 2024-04-25T08:11:27.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:11:27.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=952a3512-6dd3-4e96-aaea-a417d26c0cd1 2024-04-25T08:11:27.013 INFO:tasks.workunit.client.0.smithi102.stdout:add osd4 952a3512-6dd3-4e96-aaea-a417d26c0cd1 2024-04-25T08:11:27.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 952a3512-6dd3-4e96-aaea-a417d26c0cd1' 2024-04-25T08:11:27.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:11:27.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAvECpmZD0qAhAAhZusTa+oHsIFY9O85+RgbQ== 2024-04-25T08:11:27.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAvECpmZD0qAhAAhZusTa+oHsIFY9O85+RgbQ=="}' 2024-04-25T08:11:27.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 952a3512-6dd3-4e96-aaea-a417d26c0cd1 -i td/osd-recovery-prio/4/new.json 2024-04-25T08:11:27.856 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:11:27.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/4/new.json 2024-04-25T08:11:27.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAvECpmZD0qAhAAhZusTa+oHsIFY9O85+RgbQ== --osd-uuid 952a3512-6dd3-4e96-aaea-a417d26c0cd1 2024-04-25T08:11:27.914 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:27.911+0000 7f4cd2ef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:27.921 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:27.919+0000 7f4cd2ef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:27.922 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:27.923+0000 7f4cd2ef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:27.923 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:27.923+0000 7f4cd2ef87c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T08:11:27.923 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:27.923+0000 7f4cd2ef87c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T08:11:27.923 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:27.923+0000 7f4cd2ef87c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T08:11:27.994 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:27.995+0000 7f4cd2ef87c0 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2024-04-25T08:11:34.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2024-04-25T08:11:34.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:11:34.880 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd4 key to auth repository 2024-04-25T08:11:34.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:11:34.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:11:35.442 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.4 2024-04-25T08:11:35.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:11:35.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:11:35.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=61e7f1cf-98ad-40eb-9189-5f696423e88b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.204781/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:11:35.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:11:35.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:11:35.476 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:35.475+0000 7fb6325907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:35.487 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:35.487+0000 7fb6325907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:35.488 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:35.487+0000 7fb6325907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:11:35.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:11:35.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:11:35.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:11:35.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:11:35.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:11:35.854 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:11:35.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:35.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:11:35.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:35.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:36.072 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:36.071+0000 7fb6325907c0 -1 Falling back to public interface 2024-04-25T08:11:36.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:37.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:37.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:37.291 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:11:37.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:11:37.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:37.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:37.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:38.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:38.703 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:11:38.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:11:38.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:38.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:38.723 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:11:38.723+0000 7fb6325907c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:11:39.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:11:40.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:11:40.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:11:40.116 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:11:40.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:11:40.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:11:40.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:11:40.530 INFO:tasks.workunit.client.0.smithi102.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/774943374,v1:127.0.0.1:6835/774943374] [v2:127.0.0.1:6836/774943374,v1:127.0.0.1:6837/774943374] exists,up 952a3512-6dd3-4e96-aaea-a417d26c0cd1 2024-04-25T08:11:40.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:11:40.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:11:40.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:11:40.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: seq 1 10 2024-04-25T08:11:40.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:11:40.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test1 1 1 2024-04-25T08:11:40.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T08:11:41.307 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test1' already exists 2024-04-25T08:11:41.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:11:42.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test1 size 2 2024-04-25T08:11:43.542 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-04-25T08:11:43.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:11:43.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test2 1 1 2024-04-25T08:11:43.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-04-25T08:11:44.305 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test2' already exists 2024-04-25T08:11:44.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:11:45.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test2 size 2 2024-04-25T08:11:46.710 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 2 size to 2 2024-04-25T08:11:46.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:11:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test3 1 1 2024-04-25T08:11:46.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-04-25T08:11:47.464 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test3' already exists 2024-04-25T08:11:47.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:11:48.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-04-25T08:11:49.533 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 3 size to 2 2024-04-25T08:11:49.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:11:49.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test4 1 1 2024-04-25T08:11:49.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2024-04-25T08:11:50.260 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test4' already exists 2024-04-25T08:11:50.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:11:51.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test4 size 2 2024-04-25T08:11:52.155 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 4 size to 2 2024-04-25T08:11:52.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:11:52.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test5 1 1 2024-04-25T08:11:52.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2024-04-25T08:11:53.003 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test5' already exists 2024-04-25T08:11:53.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:11:54.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test5 size 2 2024-04-25T08:11:54.922 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 5 size to 2 2024-04-25T08:11:54.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:11:54.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test6 1 1 2024-04-25T08:11:54.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2024-04-25T08:11:55.685 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test6' already exists 2024-04-25T08:11:55.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:11:56.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test6 size 2 2024-04-25T08:11:57.664 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 6 size to 2 2024-04-25T08:11:57.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:11:57.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test7 1 1 2024-04-25T08:11:57.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2024-04-25T08:11:58.790 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test7' already exists 2024-04-25T08:11:58.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:11:59.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-04-25T08:12:00.702 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 7 size to 2 2024-04-25T08:12:00.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:00.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test8 1 1 2024-04-25T08:12:00.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2024-04-25T08:12:01.624 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test8' already exists 2024-04-25T08:12:01.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:12:02.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test8 size 2 2024-04-25T08:12:03.684 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 8 size to 2 2024-04-25T08:12:03.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:03.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test9 1 1 2024-04-25T08:12:03.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2024-04-25T08:12:04.467 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test9' already exists 2024-04-25T08:12:04.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:12:05.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test9 size 2 2024-04-25T08:12:06.375 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 9 size to 2 2024-04-25T08:12:06.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:06.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test10 1 1 2024-04-25T08:12:06.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2024-04-25T08:12:07.212 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test10' already exists 2024-04-25T08:12:07.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:12:08.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test10 size 2 2024-04-25T08:12:09.131 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 10 size to 2 2024-04-25T08:12:09.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:66: TEST_recovery_priority: sleep 5 2024-04-25T08:12:14.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:68: TEST_recovery_priority: wait_for_clean 2024-04-25T08:12:14.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:12:14.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:12:14.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:12:14.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:12:14.154 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:12:14.154 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:12:14.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:12:14.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:12:14.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:12:14.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:12:14.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:12:14.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:12:14.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:12:14.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:12:14.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:12:14.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:12:14.710 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:12:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:12:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:12:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:4' 2024-04-25T08:12:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:12:14.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:14.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:12:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836500 2024-04-25T08:12:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836500 2024-04-25T08:12:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500' 2024-04-25T08:12:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:14.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:12:15.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-04-25T08:12:15.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-04-25T08:12:15.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672978' 2024-04-25T08:12:15.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:15.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:12:15.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-04-25T08:12:15.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-04-25T08:12:15.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-64424509455' 2024-04-25T08:12:15.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:15.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:12:15.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345932 2024-04-25T08:12:15.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345932 2024-04-25T08:12:15.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-64424509455 3-85899345932' 2024-04-25T08:12:15.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:15.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:12:15.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182410 2024-04-25T08:12:15.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182410 2024-04-25T08:12:15.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672978 2-64424509455 3-85899345932 4-107374182410' 2024-04-25T08:12:15.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:15.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836500 2024-04-25T08:12:15.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:15.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:12:15.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836500 2024-04-25T08:12:15.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:15.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836500 2024-04-25T08:12:15.545 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836500 2024-04-25T08:12:15.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-04-25T08:12:15.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:12:15.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836498 -lt 21474836500 2024-04-25T08:12:15.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:12:16.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:12:16.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:12:17.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836500 2024-04-25T08:12:17.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:17.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-04-25T08:12:17.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:17.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:12:17.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-04-25T08:12:17.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:17.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-04-25T08:12:17.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-25T08:12:17.378 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672978 2024-04-25T08:12:17.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:12:17.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672978 2024-04-25T08:12:17.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:17.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:17.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-04-25T08:12:17.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:12:17.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:17.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-04-25T08:12:17.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-04-25T08:12:17.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-04-25T08:12:17.813 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509455 2024-04-25T08:12:17.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:12:18.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509455 2024-04-25T08:12:18.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:18.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345932 2024-04-25T08:12:18.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:18.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:12:18.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345932 2024-04-25T08:12:18.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:18.268 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345932 2024-04-25T08:12:18.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345932 2024-04-25T08:12:18.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-04-25T08:12:18.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:12:18.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345932 -lt 85899345932 2024-04-25T08:12:18.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:18.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182410 2024-04-25T08:12:18.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:18.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:12:18.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182410 2024-04-25T08:12:18.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:18.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182410 2024-04-25T08:12:18.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182410' 2024-04-25T08:12:18.701 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182410 2024-04-25T08:12:18.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:12:19.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182410 -lt 107374182410 2024-04-25T08:12:19.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:12:19.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:19.182 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:19.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2024-04-25T08:12:19.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:19.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:19.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:19.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:19.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:19.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:19.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:20.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2024-04-25T08:12:20.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:20.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:20.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:20.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2024-04-25T08:12:20.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:12:20.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:12:20.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:70: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:12:21.057 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:12:21.057 INFO:tasks.workunit.client.0.smithi102.stdout:10.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:12:09.887785+0000 0'0 82:17 [2,1] 2 [2,1] 2 0'0 2024-04-25T08:12:06.890008+0000 0'0 2024-04-25T08:12:06.890008+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:46:45.019785+0000 0 0 2024-04-25T08:12:21.057 INFO:tasks.workunit.client.0.smithi102.stdout:9.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:12:07.340175+0000 0'0 82:29 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:12:04.139271+0000 0'0 2024-04-25T08:12:04.139271+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:37:14.126399+0000 0 0 2024-04-25T08:12:21.057 INFO:tasks.workunit.client.0.smithi102.stdout:8.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:12:04.836374+0000 0'0 82:33 [3,4] 3 [3,4] 3 0'0 2024-04-25T08:12:01.424715+0000 0'0 2024-04-25T08:12:01.424715+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:21:59.537053+0000 0 0 2024-04-25T08:12:21.057 INFO:tasks.workunit.client.0.smithi102.stdout:7.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:12:01.844691+0000 0'0 82:55 [1,4] 1 [1,4] 1 0'0 2024-04-25T08:11:58.482060+0000 0'0 2024-04-25T08:11:58.482060+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:00:49.170149+0000 0 0 2024-04-25T08:12:21.057 INFO:tasks.workunit.client.0.smithi102.stdout:6.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:11:59.277176+0000 0'0 82:55 [4,2] 4 [4,2] 4 0'0 2024-04-25T08:11:55.497134+0000 0'0 2024-04-25T08:11:55.497134+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:21:26.278238+0000 0 0 2024-04-25T08:12:21.057 INFO:tasks.workunit.client.0.smithi102.stdout:5.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:11:55.944157+0000 0'0 82:60 [3,4] 3 [3,4] 3 0'0 2024-04-25T08:11:52.685739+0000 0'0 2024-04-25T08:11:52.685739+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:58:24.909392+0000 0 0 2024-04-25T08:12:21.057 INFO:tasks.workunit.client.0.smithi102.stdout:4.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:11:53.574940+0000 0'0 82:71 [4,0] 4 [4,0] 4 0'0 2024-04-25T08:11:49.902465+0000 0'0 2024-04-25T08:11:49.902465+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:29:40.446509+0000 0 0 2024-04-25T08:12:21.057 INFO:tasks.workunit.client.0.smithi102.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:11:50.342831+0000 0'0 82:88 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:11:47.100869+0000 0'0 2024-04-25T08:11:47.100869+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:38:52.763038+0000 0 0 2024-04-25T08:12:21.058 INFO:tasks.workunit.client.0.smithi102.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:11:47.357277+0000 0'0 82:93 [3,1] 3 [3,1] 3 0'0 2024-04-25T08:11:44.084454+0000 0'0 2024-04-25T08:11:44.084454+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:51:29.762425+0000 0 0 2024-04-25T08:12:21.058 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:11:44.668416+0000 0'0 82:113 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:11:41.048010+0000 0'0 2024-04-25T08:11:41.048010+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:26:08.996268+0000 0 0 2024-04-25T08:12:21.058 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:12:21.058 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:12:21.058 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:12:21.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local PG1 2024-04-25T08:12:21.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local POOLNUM1 2024-04-25T08:12:21.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:76: TEST_recovery_priority: local pool1 2024-04-25T08:12:21.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local chk_osd1_1 2024-04-25T08:12:21.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local chk_osd1_2 2024-04-25T08:12:21.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local PG2 2024-04-25T08:12:21.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:81: TEST_recovery_priority: local POOLNUM2 2024-04-25T08:12:21.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local pool2 2024-04-25T08:12:21.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local chk_osd2 2024-04-25T08:12:21.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:85: TEST_recovery_priority: local PG3 2024-04-25T08:12:21.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: local POOLNUM3 2024-04-25T08:12:21.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:87: TEST_recovery_priority: local pool3 2024-04-25T08:12:21.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: seq 1 10 2024-04-25T08:12:21.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:21.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 1.0 --format=json 2024-04-25T08:12:21.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:12:21.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:12:21.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T08:12:21.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:12:21.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-04-25T08:12:21.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z '' ']' 2024-04-25T08:12:21.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: PG1=1.0 2024-04-25T08:12:21.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: POOLNUM1=1 2024-04-25T08:12:21.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: pool1=test1 2024-04-25T08:12:21.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:99: TEST_recovery_priority: chk_osd1_1=1 2024-04-25T08:12:21.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: chk_osd1_2=0 2024-04-25T08:12:21.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:21.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 2.0 --format=json 2024-04-25T08:12:21.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:12:21.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:12:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-04-25T08:12:21.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:12:21.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=1 2024-04-25T08:12:21.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:12:21.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2024-04-25T08:12:21.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2024-04-25T08:12:21.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:21.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 3.0 --format=json 2024-04-25T08:12:21.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:12:22.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:12:22.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T08:12:22.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:12:22.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-04-25T08:12:22.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:12:22.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2024-04-25T08:12:22.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: PG2=3.0 2024-04-25T08:12:22.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: POOLNUM2=3 2024-04-25T08:12:22.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:105: TEST_recovery_priority: pool2=test3 2024-04-25T08:12:22.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: chk_osd2=2 2024-04-25T08:12:22.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:22.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 4.0 --format=json 2024-04-25T08:12:22.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:12:22.824 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:12:22.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-04-25T08:12:22.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:12:22.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-04-25T08:12:22.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:12:22.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2024-04-25T08:12:22.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2024-04-25T08:12:22.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:22.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 5.0 --format=json 2024-04-25T08:12:22.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:12:23.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:12:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-04-25T08:12:23.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-04-25T08:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2024-04-25T08:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-04-25T08:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 6.0 --format=json 2024-04-25T08:12:23.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:12:23.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:12:23.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-04-25T08:12:23.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:12:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-04-25T08:12:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:12:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2024-04-25T08:12:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2024-04-25T08:12:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 7.0 --format=json 2024-04-25T08:12:23.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:12:24.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:12:24.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T08:12:24.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:12:24.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-04-25T08:12:24.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:12:24.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2024-04-25T08:12:24.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-04-25T08:12:24.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: PG3=7.0 2024-04-25T08:12:24.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:110: TEST_recovery_priority: POOLNUM3=7 2024-04-25T08:12:24.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:111: TEST_recovery_priority: pool3=test7 2024-04-25T08:12:24.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: break 2024-04-25T08:12:24.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:115: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2024-04-25T08:12:24.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:117: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2024-04-25T08:12:24.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: seq 1 10 2024-04-25T08:12:24.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:24.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2024-04-25T08:12:24.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:24.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2024-04-25T08:12:24.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test2 2024-04-25T08:12:24.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-04-25T08:12:24.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-04-25T08:12:24.701 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test2' does not exist 2024-04-25T08:12:24.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:24.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2024-04-25T08:12:24.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:24.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2024-04-25T08:12:24.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test4 2024-04-25T08:12:24.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2024-04-25T08:12:24.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2024-04-25T08:12:25.536 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test4' does not exist 2024-04-25T08:12:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2024-04-25T08:12:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test5 2024-04-25T08:12:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2024-04-25T08:12:25.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2024-04-25T08:12:26.127 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test5' does not exist 2024-04-25T08:12:26.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:26.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2024-04-25T08:12:26.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test6 2024-04-25T08:12:26.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2024-04-25T08:12:26.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2024-04-25T08:12:26.737 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test6' does not exist 2024-04-25T08:12:26.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:26.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2024-04-25T08:12:26.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:26.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2024-04-25T08:12:26.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test8 2024-04-25T08:12:26.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2024-04-25T08:12:26.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2024-04-25T08:12:27.264 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test8' does not exist 2024-04-25T08:12:27.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:27.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2024-04-25T08:12:27.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test9 2024-04-25T08:12:27.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2024-04-25T08:12:27.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2024-04-25T08:12:27.783 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test9' does not exist 2024-04-25T08:12:27.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:12:27.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2024-04-25T08:12:27.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test10 2024-04-25T08:12:27.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2024-04-25T08:12:27.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2024-04-25T08:12:28.619 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test10' does not exist 2024-04-25T08:12:28.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:131: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2024-04-25T08:12:30.012 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 3 size to 1 2024-04-25T08:12:30.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2024-04-25T08:12:31.010 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 7 size to 1 2024-04-25T08:12:31.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: wait_for_clean 2024-04-25T08:12:31.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:12:31.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:12:31.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:12:31.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:12:31.037 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:12:31.037 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:12:31.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:12:31.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:12:31.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:12:31.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:12:31.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:12:31.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:12:31.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:12:31.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:12:31.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:12:31.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:12:31.711 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:12:31.711 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:12:31.711 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:12:31.711 INFO:tasks.workunit.client.0.smithi102.stderr:4' 2024-04-25T08:12:31.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:12:31.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:31.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:12:31.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836506 2024-04-25T08:12:31.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836506 2024-04-25T08:12:31.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506' 2024-04-25T08:12:31.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:31.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:12:32.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672983 2024-04-25T08:12:32.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672983 2024-04-25T08:12:32.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672983' 2024-04-25T08:12:32.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:32.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:12:32.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509460 2024-04-25T08:12:32.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509460 2024-04-25T08:12:32.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672983 2-64424509460' 2024-04-25T08:12:32.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:32.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:12:32.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345938 2024-04-25T08:12:32.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345938 2024-04-25T08:12:32.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672983 2-64424509460 3-85899345938' 2024-04-25T08:12:32.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:32.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:12:32.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182415 2024-04-25T08:12:32.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182415 2024-04-25T08:12:32.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672983 2-64424509460 3-85899345938 4-107374182415' 2024-04-25T08:12:32.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:32.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836506 2024-04-25T08:12:32.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:32.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:12:32.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-04-25T08:12:32.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:32.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836506 2024-04-25T08:12:32.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-04-25T08:12:32.515 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836506 2024-04-25T08:12:32.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:12:32.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836506 2024-04-25T08:12:32.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:32.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672983 2024-04-25T08:12:32.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:32.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:12:32.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-04-25T08:12:32.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:32.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672983 2024-04-25T08:12:32.944 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672983 2024-04-25T08:12:32.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-04-25T08:12:32.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:12:33.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672983 -lt 42949672983 2024-04-25T08:12:33.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:33.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509460 2024-04-25T08:12:33.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:33.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:12:33.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:33.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509460 2024-04-25T08:12:33.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509460 2024-04-25T08:12:33.419 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509460 2024-04-25T08:12:33.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-04-25T08:12:33.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:12:33.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509460 -lt 64424509460 2024-04-25T08:12:33.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:33.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345938 2024-04-25T08:12:33.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:33.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:12:33.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345938 2024-04-25T08:12:33.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:33.877 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345938 2024-04-25T08:12:33.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345938 2024-04-25T08:12:33.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345938' 2024-04-25T08:12:33.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:12:34.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345938 -lt 85899345938 2024-04-25T08:12:34.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:34.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182415 2024-04-25T08:12:34.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:34.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:12:34.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182415 2024-04-25T08:12:34.298 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:34.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182415 2024-04-25T08:12:34.300 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182415 2024-04-25T08:12:34.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-04-25T08:12:34.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:12:34.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182415 -lt 107374182415 2024-04-25T08:12:34.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:12:34.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:34.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:35.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T08:12:35.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:35.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:35.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:35.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:35.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:35.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:35.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:35.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T08:12:35.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:35.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:35.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:36.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T08:12:36.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:12:36.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:12:36.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:135: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-04-25T08:12:36.269 INFO:tasks.workunit.client.0.smithi102.stderr:10+0 records in 2024-04-25T08:12:36.269 INFO:tasks.workunit.client.0.smithi102.stderr:10+0 records out 2024-04-25T08:12:36.270 INFO:tasks.workunit.client.0.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0384999 s, 272 MB/s 2024-04-25T08:12:36.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: p=1 2024-04-25T08:12:36.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T08:12:36.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T08:12:36.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:36.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-04-25T08:12:37.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:37.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-04-25T08:12:37.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:37.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-04-25T08:12:38.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:38.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-04-25T08:12:39.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:39.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-04-25T08:12:40.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:40.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-04-25T08:12:40.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:40.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-04-25T08:12:41.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:41.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-04-25T08:12:42.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:42.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-04-25T08:12:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-04-25T08:12:43.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:43.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-04-25T08:12:44.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:44.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-04-25T08:12:45.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:45.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-04-25T08:12:45.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:45.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-04-25T08:12:46.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:46.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-04-25T08:12:47.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:47.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-04-25T08:12:48.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:48.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-04-25T08:12:48.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:48.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-04-25T08:12:49.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:49.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-04-25T08:12:50.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:50.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-04-25T08:12:51.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:51.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-04-25T08:12:51.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:51.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-04-25T08:12:52.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:52.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-04-25T08:12:53.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:53.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-04-25T08:12:54.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:54.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-04-25T08:12:54.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:54.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-04-25T08:12:55.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:55.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-04-25T08:12:56.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:56.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-04-25T08:12:57.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:57.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-04-25T08:12:57.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:57.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-04-25T08:12:58.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:58.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-04-25T08:12:59.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:59.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-04-25T08:13:00.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:00.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-04-25T08:13:01.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:01.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-04-25T08:13:01.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:01.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-04-25T08:13:02.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:02.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-04-25T08:13:03.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:03.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-04-25T08:13:04.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:04.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-04-25T08:13:04.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:04.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-04-25T08:13:05.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:05.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-04-25T08:13:06.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:06.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-04-25T08:13:07.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:07.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-04-25T08:13:08.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:08.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-04-25T08:13:08.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:08.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-04-25T08:13:09.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:09.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-04-25T08:13:10.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:10.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-04-25T08:13:11.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:11.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-04-25T08:13:12.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:12.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-04-25T08:13:12.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:12.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-04-25T08:13:13.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:13.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-04-25T08:13:14.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:14.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-04-25T08:13:15.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:15.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-04-25T08:13:16.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:16.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-04-25T08:13:16.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:16.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-04-25T08:13:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:17.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-04-25T08:13:18.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:18.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-04-25T08:13:19.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:19.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-04-25T08:13:19.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:19.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-04-25T08:13:20.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:20.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-04-25T08:13:21.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:21.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-04-25T08:13:22.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:22.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-04-25T08:13:23.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:23.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-04-25T08:13:23.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:23.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-04-25T08:13:24.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:24.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-04-25T08:13:25.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:25.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-04-25T08:13:26.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:26.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-04-25T08:13:27.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:27.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-04-25T08:13:27.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:27.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-04-25T08:13:28.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:28.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-04-25T08:13:29.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:29.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-04-25T08:13:30.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:30.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-04-25T08:13:30.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:30.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-04-25T08:13:31.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:31.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-04-25T08:13:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:32.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-04-25T08:13:33.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:33.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-04-25T08:13:34.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:34.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-04-25T08:13:34.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:34.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-04-25T08:13:35.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:35.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-04-25T08:13:36.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:36.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-04-25T08:13:37.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:37.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-04-25T08:13:37.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:37.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-04-25T08:13:38.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:38.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-04-25T08:13:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:39.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-04-25T08:13:40.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:40.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-04-25T08:13:40.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:40.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-04-25T08:13:41.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:41.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-04-25T08:13:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-04-25T08:13:43.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:43.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-04-25T08:13:43.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:43.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-04-25T08:13:44.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:44.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-04-25T08:13:45.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:45.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-04-25T08:13:46.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:46.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-04-25T08:13:46.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:46.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-04-25T08:13:47.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:47.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-04-25T08:13:48.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:48.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-04-25T08:13:49.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:49.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-04-25T08:13:50.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:50.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-04-25T08:13:50.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:50.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-04-25T08:13:51.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:51.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-04-25T08:13:52.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:52.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-04-25T08:13:53.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:53.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-04-25T08:13:54.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:54.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-04-25T08:13:54.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:54.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-04-25T08:13:55.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:55.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-04-25T08:13:56.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:56.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-04-25T08:13:57.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:57.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-04-25T08:13:57.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:57.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-04-25T08:13:58.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:58.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-04-25T08:13:59.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:13:59.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-04-25T08:14:00.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:00.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-04-25T08:14:01.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:01.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-04-25T08:14:02.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:02.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-04-25T08:14:02.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:02.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-04-25T08:14:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-04-25T08:14:04.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:04.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-04-25T08:14:05.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:05.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-04-25T08:14:05.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:05.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-04-25T08:14:06.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:06.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-04-25T08:14:07.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:07.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-04-25T08:14:08.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:08.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-04-25T08:14:08.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:08.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-04-25T08:14:09.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:09.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-04-25T08:14:10.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:10.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-04-25T08:14:11.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:11.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-04-25T08:14:12.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:12.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-04-25T08:14:12.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:12.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-04-25T08:14:13.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:13.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-04-25T08:14:14.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:14.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-04-25T08:14:15.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:15.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-04-25T08:14:15.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:15.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-04-25T08:14:16.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:16.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-04-25T08:14:17.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:17.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-04-25T08:14:18.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:18.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-04-25T08:14:18.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:18.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-04-25T08:14:19.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:19.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-04-25T08:14:20.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:20.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-04-25T08:14:21.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:21.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-04-25T08:14:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:22.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-04-25T08:14:22.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:22.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-04-25T08:14:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-04-25T08:14:24.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:24.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-04-25T08:14:25.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:25.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-04-25T08:14:25.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:25.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-04-25T08:14:26.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:26.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-04-25T08:14:27.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:27.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-04-25T08:14:28.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:28.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-04-25T08:14:28.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:28.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-04-25T08:14:29.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:29.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-04-25T08:14:30.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:30.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-04-25T08:14:31.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:31.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-04-25T08:14:31.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:31.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-04-25T08:14:32.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:32.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-04-25T08:14:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:33.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-04-25T08:14:34.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:34.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-04-25T08:14:35.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:35.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-04-25T08:14:36.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:36.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-04-25T08:14:36.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:36.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-04-25T08:14:37.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:37.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-04-25T08:14:38.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:38.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-04-25T08:14:39.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:39.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-04-25T08:14:39.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:39.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-04-25T08:14:40.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:40.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-04-25T08:14:41.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:41.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-04-25T08:14:42.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:42.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-04-25T08:14:43.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:43.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-04-25T08:14:43.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:43.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-04-25T08:14:44.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:44.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-04-25T08:14:45.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:45.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-04-25T08:14:46.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:46.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-04-25T08:14:46.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:46.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-04-25T08:14:47.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:47.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-04-25T08:14:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:48.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-04-25T08:14:49.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:49.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-04-25T08:14:49.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:49.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-04-25T08:14:50.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:50.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-04-25T08:14:51.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:51.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-04-25T08:14:52.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:52.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-04-25T08:14:52.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:52.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-04-25T08:14:53.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:53.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-04-25T08:14:54.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:54.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-04-25T08:14:55.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:55.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-04-25T08:14:55.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:55.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-04-25T08:14:56.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:56.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-04-25T08:14:57.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:57.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-04-25T08:14:58.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:58.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-04-25T08:14:58.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:58.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-04-25T08:14:59.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:14:59.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-04-25T08:15:00.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:00.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-04-25T08:15:01.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:01.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-04-25T08:15:01.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:01.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-04-25T08:15:02.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:02.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-04-25T08:15:03.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:03.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-04-25T08:15:04.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:04.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-04-25T08:15:05.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:05.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-04-25T08:15:05.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:05.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-04-25T08:15:06.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:06.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-04-25T08:15:07.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:07.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-04-25T08:15:08.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:08.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-04-25T08:15:08.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:08.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-04-25T08:15:09.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:09.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-04-25T08:15:10.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 1 + 1 2024-04-25T08:15:10.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=2 2024-04-25T08:15:10.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T08:15:10.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T08:15:10.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:10.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2024-04-25T08:15:10.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:10.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2024-04-25T08:15:11.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:11.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2024-04-25T08:15:11.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:11.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2024-04-25T08:15:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:12.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2024-04-25T08:15:12.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:12.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2024-04-25T08:15:12.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:12.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2024-04-25T08:15:13.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:13.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2024-04-25T08:15:13.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:13.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2024-04-25T08:15:14.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:14.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2024-04-25T08:15:14.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:14.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2024-04-25T08:15:15.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:15.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2024-04-25T08:15:15.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:15.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2024-04-25T08:15:15.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:15.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2024-04-25T08:15:16.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:16.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2024-04-25T08:15:16.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:16.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2024-04-25T08:15:17.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:17.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2024-04-25T08:15:17.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:17.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2024-04-25T08:15:18.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:18.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2024-04-25T08:15:18.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:18.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2024-04-25T08:15:18.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:18.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2024-04-25T08:15:19.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:19.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2024-04-25T08:15:19.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:19.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2024-04-25T08:15:20.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:20.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2024-04-25T08:15:20.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:20.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2024-04-25T08:15:21.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:21.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2024-04-25T08:15:21.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:21.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2024-04-25T08:15:21.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:21.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2024-04-25T08:15:22.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:22.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2024-04-25T08:15:22.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:22.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2024-04-25T08:15:23.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:23.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2024-04-25T08:15:23.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:23.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2024-04-25T08:15:23.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:23.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2024-04-25T08:15:24.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:24.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2024-04-25T08:15:24.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:24.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2024-04-25T08:15:25.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:25.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2024-04-25T08:15:25.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:25.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2024-04-25T08:15:26.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:26.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2024-04-25T08:15:26.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:26.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2024-04-25T08:15:26.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:26.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2024-04-25T08:15:27.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:27.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2024-04-25T08:15:27.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:27.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2024-04-25T08:15:28.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:28.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2024-04-25T08:15:28.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:28.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2024-04-25T08:15:29.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:29.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2024-04-25T08:15:29.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:29.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2024-04-25T08:15:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2024-04-25T08:15:30.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:30.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2024-04-25T08:15:30.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:30.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2024-04-25T08:15:31.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:31.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2024-04-25T08:15:31.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:31.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2024-04-25T08:15:32.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:32.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2024-04-25T08:15:32.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:32.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2024-04-25T08:15:33.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:33.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2024-04-25T08:15:33.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:33.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2024-04-25T08:15:33.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:33.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2024-04-25T08:15:34.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:34.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2024-04-25T08:15:35.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:35.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2024-04-25T08:15:35.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:35.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2024-04-25T08:15:35.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:35.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2024-04-25T08:15:36.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:36.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2024-04-25T08:15:36.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:36.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2024-04-25T08:15:37.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:37.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2024-04-25T08:15:37.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:37.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2024-04-25T08:15:38.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:38.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2024-04-25T08:15:38.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:38.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2024-04-25T08:15:38.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:38.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2024-04-25T08:15:39.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:39.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2024-04-25T08:15:39.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:39.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2024-04-25T08:15:40.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2024-04-25T08:15:40.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:40.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2024-04-25T08:15:41.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:41.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2024-04-25T08:15:41.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:41.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2024-04-25T08:15:41.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:41.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2024-04-25T08:15:42.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:42.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2024-04-25T08:15:42.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:42.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2024-04-25T08:15:43.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:43.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2024-04-25T08:15:43.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:43.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2024-04-25T08:15:43.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:43.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2024-04-25T08:15:44.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:44.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2024-04-25T08:15:44.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:44.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2024-04-25T08:15:45.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:45.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2024-04-25T08:15:45.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:45.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2024-04-25T08:15:46.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:46.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2024-04-25T08:15:46.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:46.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2024-04-25T08:15:46.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:46.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2024-04-25T08:15:47.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:47.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2024-04-25T08:15:47.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:47.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2024-04-25T08:15:48.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:48.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2024-04-25T08:15:48.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:48.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2024-04-25T08:15:48.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:48.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2024-04-25T08:15:49.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:49.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2024-04-25T08:15:49.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:49.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2024-04-25T08:15:50.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:50.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2024-04-25T08:15:50.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:50.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2024-04-25T08:15:50.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:50.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2024-04-25T08:15:51.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:51.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2024-04-25T08:15:51.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:51.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2024-04-25T08:15:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:52.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2024-04-25T08:15:52.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:52.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2024-04-25T08:15:53.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:53.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2024-04-25T08:15:53.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:53.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2024-04-25T08:15:54.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:54.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2024-04-25T08:15:54.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:54.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2024-04-25T08:15:54.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:54.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2024-04-25T08:15:55.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:55.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2024-04-25T08:15:55.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:55.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2024-04-25T08:15:56.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:56.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2024-04-25T08:15:56.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:56.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2024-04-25T08:15:57.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:57.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2024-04-25T08:15:57.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:57.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2024-04-25T08:15:58.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:58.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2024-04-25T08:15:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:58.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2024-04-25T08:15:58.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:58.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2024-04-25T08:15:59.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:59.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2024-04-25T08:15:59.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:15:59.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2024-04-25T08:16:00.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:00.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2024-04-25T08:16:00.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:00.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2024-04-25T08:16:01.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:01.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2024-04-25T08:16:01.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:01.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2024-04-25T08:16:01.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:01.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2024-04-25T08:16:02.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:02.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2024-04-25T08:16:02.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:02.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2024-04-25T08:16:03.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:03.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2024-04-25T08:16:03.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:03.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2024-04-25T08:16:04.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:04.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2024-04-25T08:16:04.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:04.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2024-04-25T08:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:05.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2024-04-25T08:16:05.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:05.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2024-04-25T08:16:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2024-04-25T08:16:06.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:06.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2024-04-25T08:16:07.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:07.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2024-04-25T08:16:07.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:07.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2024-04-25T08:16:07.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:07.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2024-04-25T08:16:08.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:08.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2024-04-25T08:16:08.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:08.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2024-04-25T08:16:09.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:09.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2024-04-25T08:16:09.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:09.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2024-04-25T08:16:10.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:10.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2024-04-25T08:16:10.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:10.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2024-04-25T08:16:10.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:10.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2024-04-25T08:16:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:11.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2024-04-25T08:16:11.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:11.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2024-04-25T08:16:12.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:12.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2024-04-25T08:16:12.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:12.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2024-04-25T08:16:13.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:13.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2024-04-25T08:16:13.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:13.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2024-04-25T08:16:14.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:14.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2024-04-25T08:16:14.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:14.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2024-04-25T08:16:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:14.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2024-04-25T08:16:15.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:15.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2024-04-25T08:16:15.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:15.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2024-04-25T08:16:16.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:16.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2024-04-25T08:16:16.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:16.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2024-04-25T08:16:16.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:16.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2024-04-25T08:16:17.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:17.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2024-04-25T08:16:17.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:17.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2024-04-25T08:16:18.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:18.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2024-04-25T08:16:18.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:18.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2024-04-25T08:16:19.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:19.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2024-04-25T08:16:19.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:19.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2024-04-25T08:16:19.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:19.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2024-04-25T08:16:20.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:20.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2024-04-25T08:16:20.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:20.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2024-04-25T08:16:21.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:21.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2024-04-25T08:16:21.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:21.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2024-04-25T08:16:22.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:22.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2024-04-25T08:16:22.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:22.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2024-04-25T08:16:22.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:22.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2024-04-25T08:16:23.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:23.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2024-04-25T08:16:23.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:23.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2024-04-25T08:16:24.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:24.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2024-04-25T08:16:24.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:24.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2024-04-25T08:16:25.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:25.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2024-04-25T08:16:25.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:25.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2024-04-25T08:16:25.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:25.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2024-04-25T08:16:26.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:26.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2024-04-25T08:16:26.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:26.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2024-04-25T08:16:27.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:27.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2024-04-25T08:16:27.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:27.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2024-04-25T08:16:28.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:28.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2024-04-25T08:16:28.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:28.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2024-04-25T08:16:28.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:28.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2024-04-25T08:16:29.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:29.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2024-04-25T08:16:29.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:29.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2024-04-25T08:16:30.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:30.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2024-04-25T08:16:30.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:30.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2024-04-25T08:16:30.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:30.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2024-04-25T08:16:31.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:31.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2024-04-25T08:16:31.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:31.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2024-04-25T08:16:32.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:32.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2024-04-25T08:16:32.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:32.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2024-04-25T08:16:33.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:33.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2024-04-25T08:16:33.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:33.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2024-04-25T08:16:34.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:34.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2024-04-25T08:16:34.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:34.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2024-04-25T08:16:34.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:34.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2024-04-25T08:16:35.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:35.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2024-04-25T08:16:35.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:35.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2024-04-25T08:16:36.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:36.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2024-04-25T08:16:36.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 2 + 1 2024-04-25T08:16:36.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=3 2024-04-25T08:16:36.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T08:16:36.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T08:16:36.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:36.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2024-04-25T08:16:37.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:37.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2024-04-25T08:16:37.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:37.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2024-04-25T08:16:38.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:38.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2024-04-25T08:16:38.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:38.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2024-04-25T08:16:39.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:39.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2024-04-25T08:16:39.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:39.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2024-04-25T08:16:40.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:40.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2024-04-25T08:16:40.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:40.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2024-04-25T08:16:40.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:40.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2024-04-25T08:16:41.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:41.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2024-04-25T08:16:41.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:41.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2024-04-25T08:16:42.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:42.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2024-04-25T08:16:42.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:42.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2024-04-25T08:16:43.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:43.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2024-04-25T08:16:43.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:43.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2024-04-25T08:16:44.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:44.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2024-04-25T08:16:44.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:44.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2024-04-25T08:16:44.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:44.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2024-04-25T08:16:45.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:45.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2024-04-25T08:16:45.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:45.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2024-04-25T08:16:46.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:46.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2024-04-25T08:16:46.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:46.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2024-04-25T08:16:47.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:47.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2024-04-25T08:16:47.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:47.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2024-04-25T08:16:48.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:48.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2024-04-25T08:16:48.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:48.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2024-04-25T08:16:49.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:49.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2024-04-25T08:16:49.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:49.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2024-04-25T08:16:50.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:50.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2024-04-25T08:16:50.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:50.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2024-04-25T08:16:50.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:50.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2024-04-25T08:16:51.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:51.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2024-04-25T08:16:51.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:51.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2024-04-25T08:16:52.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:52.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2024-04-25T08:16:52.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:52.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2024-04-25T08:16:53.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:53.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2024-04-25T08:16:53.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:53.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2024-04-25T08:16:54.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:54.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2024-04-25T08:16:54.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:54.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2024-04-25T08:16:55.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:55.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2024-04-25T08:16:55.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:55.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2024-04-25T08:16:55.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:55.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2024-04-25T08:16:56.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:56.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2024-04-25T08:16:56.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:56.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2024-04-25T08:16:57.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:57.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2024-04-25T08:16:57.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:57.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2024-04-25T08:16:58.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:58.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2024-04-25T08:16:58.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:58.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2024-04-25T08:16:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2024-04-25T08:16:59.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:16:59.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2024-04-25T08:17:00.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:00.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2024-04-25T08:17:00.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:00.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2024-04-25T08:17:00.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:00.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2024-04-25T08:17:01.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:01.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2024-04-25T08:17:01.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:01.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2024-04-25T08:17:02.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:02.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2024-04-25T08:17:02.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:02.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2024-04-25T08:17:03.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:03.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2024-04-25T08:17:03.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:03.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2024-04-25T08:17:04.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:04.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2024-04-25T08:17:04.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:04.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2024-04-25T08:17:05.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:05.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2024-04-25T08:17:05.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:05.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2024-04-25T08:17:05.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:05.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2024-04-25T08:17:06.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:06.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2024-04-25T08:17:06.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:06.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2024-04-25T08:17:07.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:07.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2024-04-25T08:17:08.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:08.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2024-04-25T08:17:08.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:08.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2024-04-25T08:17:09.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:09.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2024-04-25T08:17:09.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:09.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2024-04-25T08:17:10.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:10.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2024-04-25T08:17:10.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:10.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2024-04-25T08:17:10.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:10.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2024-04-25T08:17:11.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:11.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2024-04-25T08:17:11.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:11.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2024-04-25T08:17:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2024-04-25T08:17:12.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:12.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2024-04-25T08:17:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:13.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2024-04-25T08:17:13.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:13.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2024-04-25T08:17:14.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:14.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2024-04-25T08:17:14.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:14.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2024-04-25T08:17:15.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:15.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2024-04-25T08:17:15.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:15.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2024-04-25T08:17:16.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:16.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2024-04-25T08:17:16.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:16.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2024-04-25T08:17:17.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:17.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2024-04-25T08:17:17.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:17.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2024-04-25T08:17:18.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:18.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2024-04-25T08:17:18.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:18.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2024-04-25T08:17:18.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:18.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2024-04-25T08:17:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2024-04-25T08:17:19.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:19.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2024-04-25T08:17:20.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:20.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2024-04-25T08:17:20.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:20.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2024-04-25T08:17:21.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:21.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2024-04-25T08:17:21.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:21.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2024-04-25T08:17:22.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:22.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2024-04-25T08:17:22.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:22.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2024-04-25T08:17:23.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:23.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2024-04-25T08:17:23.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:23.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2024-04-25T08:17:23.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:23.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2024-04-25T08:17:24.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:24.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2024-04-25T08:17:24.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:24.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2024-04-25T08:17:25.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:25.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2024-04-25T08:17:25.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:25.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2024-04-25T08:17:26.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:26.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2024-04-25T08:17:26.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:26.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2024-04-25T08:17:27.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:27.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2024-04-25T08:17:27.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:27.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2024-04-25T08:17:28.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:28.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2024-04-25T08:17:28.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:28.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2024-04-25T08:17:29.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:29.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2024-04-25T08:17:29.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:29.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2024-04-25T08:17:29.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:29.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2024-04-25T08:17:30.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:30.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2024-04-25T08:17:30.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:30.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2024-04-25T08:17:31.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:31.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2024-04-25T08:17:31.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:31.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2024-04-25T08:17:32.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:32.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2024-04-25T08:17:32.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:32.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2024-04-25T08:17:33.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:33.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2024-04-25T08:17:33.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:33.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2024-04-25T08:17:34.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2024-04-25T08:17:34.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:34.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2024-04-25T08:17:34.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:34.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2024-04-25T08:17:35.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:35.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2024-04-25T08:17:35.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:35.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2024-04-25T08:17:36.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2024-04-25T08:17:36.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:36.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2024-04-25T08:17:37.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:37.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2024-04-25T08:17:37.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:37.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2024-04-25T08:17:38.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:38.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2024-04-25T08:17:38.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:38.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2024-04-25T08:17:39.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:39.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2024-04-25T08:17:39.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:39.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2024-04-25T08:17:40.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:40.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2024-04-25T08:17:40.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:40.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2024-04-25T08:17:41.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:41.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2024-04-25T08:17:41.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:41.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2024-04-25T08:17:42.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:42.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2024-04-25T08:17:42.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:42.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2024-04-25T08:17:43.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:43.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2024-04-25T08:17:43.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:43.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2024-04-25T08:17:44.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:44.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2024-04-25T08:17:44.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:44.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2024-04-25T08:17:44.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:44.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2024-04-25T08:17:45.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:45.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2024-04-25T08:17:45.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:45.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2024-04-25T08:17:46.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:46.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2024-04-25T08:17:46.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:46.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2024-04-25T08:17:47.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:47.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2024-04-25T08:17:47.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:47.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2024-04-25T08:17:48.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:48.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2024-04-25T08:17:48.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:48.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2024-04-25T08:17:49.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:49.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2024-04-25T08:17:49.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:49.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2024-04-25T08:17:49.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:49.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2024-04-25T08:17:50.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:50.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2024-04-25T08:17:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:50.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2024-04-25T08:17:51.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:51.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2024-04-25T08:17:51.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:51.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2024-04-25T08:17:52.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:52.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2024-04-25T08:17:52.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:52.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2024-04-25T08:17:53.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:53.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2024-04-25T08:17:53.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:53.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2024-04-25T08:17:54.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:54.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2024-04-25T08:17:54.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:54.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2024-04-25T08:17:54.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:54.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2024-04-25T08:17:55.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:55.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2024-04-25T08:17:55.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:55.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2024-04-25T08:17:56.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:56.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2024-04-25T08:17:56.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:56.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2024-04-25T08:17:57.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:57.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2024-04-25T08:17:57.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:57.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2024-04-25T08:17:58.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:58.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2024-04-25T08:17:58.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:58.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2024-04-25T08:17:58.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:58.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2024-04-25T08:17:59.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:59.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2024-04-25T08:17:59.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:17:59.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2024-04-25T08:18:00.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:00.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2024-04-25T08:18:00.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:00.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2024-04-25T08:18:01.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:01.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2024-04-25T08:18:01.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:01.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2024-04-25T08:18:02.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:02.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2024-04-25T08:18:02.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:02.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2024-04-25T08:18:02.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:02.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2024-04-25T08:18:03.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:03.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2024-04-25T08:18:03.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:03.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2024-04-25T08:18:04.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:04.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2024-04-25T08:18:04.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:04.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2024-04-25T08:18:05.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:05.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2024-04-25T08:18:05.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:05.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2024-04-25T08:18:06.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:06.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2024-04-25T08:18:06.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:06.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2024-04-25T08:18:06.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:06.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2024-04-25T08:18:07.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:07.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2024-04-25T08:18:07.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:07.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2024-04-25T08:18:08.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:08.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2024-04-25T08:18:08.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 3 + 1 2024-04-25T08:18:08.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=4 2024-04-25T08:18:08.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: get_not_primary test1 obj1-p1 2024-04-25T08:18:08.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-04-25T08:18:08.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-04-25T08:18:08.706 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-04-25T08:18:08.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-04-25T08:18:08.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-04-25T08:18:08.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:18:08.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:18:09.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:18:09.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:18:09.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:18:09.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: local otherosd=0 2024-04-25T08:18:09.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:18:10.235 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:18:10.235 INFO:tasks.workunit.client.0.smithi102.stdout:7.0 191 0 0 0 0 2000683008 0 0 572 0 572 active+clean 2024-04-25T08:12:31.238003+0000 93'572 93:653 [1] 1 [1] 1 0'0 2024-04-25T08:11:58.482060+0000 0'0 2024-04-25T08:11:58.482060+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:02:24.479873+0000 0 0 2024-04-25T08:18:10.235 INFO:tasks.workunit.client.0.smithi102.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:12:30.909558+0000 93'600 93:713 [1] 1 [1] 1 0'0 2024-04-25T08:11:47.100869+0000 0'0 2024-04-25T08:11:47.100869+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:41:57.489743+0000 0 0 2024-04-25T08:18:10.235 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:11:44.668416+0000 93'600 93:735 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:11:41.048010+0000 0'0 2024-04-25T08:11:41.048010+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:26:08.996268+0000 0 0 2024-04-25T08:18:10.235 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:18:10.236 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:18:10.236 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:18:10.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ERRORS=0 2024-04-25T08:18:10.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:151: TEST_recovery_priority: ceph osd set norecover 2024-04-25T08:18:11.028 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is set 2024-04-25T08:18:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:152: TEST_recovery_priority: ceph osd set noout 2024-04-25T08:18:11.992 INFO:tasks.workunit.client.0.smithi102.stderr:noout is set 2024-04-25T08:18:12.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:156: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-04-25T08:18:12.703 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 7 size to 2 2024-04-25T08:18:12.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:157: TEST_recovery_priority: sleep 2 2024-04-25T08:18:14.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:18:14.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:18:14.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:18:14.729 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:18:14.729 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:14.730 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:18:14.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.1.asok 2024-04-25T08:18:14.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:18:14.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:14.850 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:18:14.850 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:18:14.850 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:14.850 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:14.850 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:18:14.850 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:18:14.850 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:18:14.850 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:18:14.850 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:14.850 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:14.851 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:18:14.851 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:18:14.851 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:14.851 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:18:14.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: seq 1 10 2024-04-25T08:18:14.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T08:18:14.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T08:18:14.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T08:18:15.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 1 = 10 ']' 2024-04-25T08:18:15.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-04-25T08:18:17.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T08:18:17.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T08:18:17.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T08:18:17.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 2 = 10 ']' 2024-04-25T08:18:17.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-04-25T08:18:19.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T08:18:19.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T08:18:19.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T08:18:20.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 3 = 10 ']' 2024-04-25T08:18:20.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-04-25T08:18:22.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T08:18:22.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T08:18:22.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T08:18:22.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:164: TEST_recovery_priority: break 2024-04-25T08:18:22.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:18:22.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:18:22.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:18:22.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:18:22.949 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:18:22.949 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:18:22.949 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:18:22.949 INFO:tasks.workunit.client.0.smithi102.stderr:4' 2024-04-25T08:18:22.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:18:22.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:22.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:18:23.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836578 2024-04-25T08:18:23.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836578 2024-04-25T08:18:23.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836578' 2024-04-25T08:18:23.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:23.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:18:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673055 2024-04-25T08:18:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673055 2024-04-25T08:18:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836578 1-42949673055' 2024-04-25T08:18:23.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:23.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:18:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509532 2024-04-25T08:18:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509532 2024-04-25T08:18:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836578 1-42949673055 2-64424509532' 2024-04-25T08:18:23.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:23.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:18:23.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346010 2024-04-25T08:18:23.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346010 2024-04-25T08:18:23.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836578 1-42949673055 2-64424509532 3-85899346010' 2024-04-25T08:18:23.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:23.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:18:23.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182487 2024-04-25T08:18:23.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182487 2024-04-25T08:18:23.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836578 1-42949673055 2-64424509532 3-85899346010 4-107374182487' 2024-04-25T08:18:23.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:23.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836578 2024-04-25T08:18:23.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:23.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:18:23.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836578 2024-04-25T08:18:23.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:23.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836578 2024-04-25T08:18:23.762 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836578 2024-04-25T08:18:23.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836578' 2024-04-25T08:18:23.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:18:24.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836576 -lt 21474836578 2024-04-25T08:18:24.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:18:25.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:18:25.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:18:25.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836578 -lt 21474836578 2024-04-25T08:18:25.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:25.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673055 2024-04-25T08:18:25.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:25.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:18:25.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673055 2024-04-25T08:18:25.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:25.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673055 2024-04-25T08:18:25.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673055' 2024-04-25T08:18:25.601 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673055 2024-04-25T08:18:25.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:18:26.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673056 -lt 42949673055 2024-04-25T08:18:26.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:26.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509532 2024-04-25T08:18:26.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:26.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:18:26.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509532 2024-04-25T08:18:26.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:26.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509532 2024-04-25T08:18:26.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509532' 2024-04-25T08:18:26.027 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509532 2024-04-25T08:18:26.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:18:26.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509533 -lt 64424509532 2024-04-25T08:18:26.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:26.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346010 2024-04-25T08:18:26.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:26.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:18:26.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346010 2024-04-25T08:18:26.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:26.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346010 2024-04-25T08:18:26.465 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899346010 2024-04-25T08:18:26.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346010' 2024-04-25T08:18:26.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:18:26.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346010 -lt 85899346010 2024-04-25T08:18:26.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:26.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182487 2024-04-25T08:18:26.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:26.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:18:26.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182487 2024-04-25T08:18:26.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:26.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182487 2024-04-25T08:18:26.888 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182487 2024-04-25T08:18:26.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182487' 2024-04-25T08:18:26.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:18:27.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182488 -lt 107374182487 2024-04-25T08:18:27.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:18:27.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:18:27.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:18:27.359 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:18:27.359 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:27.359 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:18:27.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.1.asok 2024-04-25T08:18:27.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:18:27.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:27.467 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:18:27.467 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:18:27.467 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:27.467 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:27.467 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:18:27.467 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:18:27.467 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:27.467 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:18:27.467 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:18:27.468 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:27.468 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:27.468 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:27.468 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:18:27.468 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:18:27.468 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:27.468 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:27.468 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:18:27.468 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:18:27.468 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:27.468 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:18:27.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph osd out osd.0 2024-04-25T08:18:28.001 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. 2024-04-25T08:18:28.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: sleep 2 2024-04-25T08:18:30.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:177: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:18:30.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:18:30.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:18:30.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:18:30.463 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:18:30.463 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:18:30.463 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:18:30.463 INFO:tasks.workunit.client.0.smithi102.stderr:4' 2024-04-25T08:18:30.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:18:30.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:30.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:18:30.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836581 2024-04-25T08:18:30.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836581 2024-04-25T08:18:30.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581' 2024-04-25T08:18:30.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:30.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:18:30.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673059 2024-04-25T08:18:30.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673059 2024-04-25T08:18:30.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673059' 2024-04-25T08:18:30.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:30.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:18:30.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509536 2024-04-25T08:18:30.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509536 2024-04-25T08:18:30.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673059 2-64424509536' 2024-04-25T08:18:30.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:30.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:18:31.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346013 2024-04-25T08:18:31.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346013 2024-04-25T08:18:31.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673059 2-64424509536 3-85899346013' 2024-04-25T08:18:31.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:31.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:18:31.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182491 2024-04-25T08:18:31.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182491 2024-04-25T08:18:31.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836581 1-42949673059 2-64424509536 3-85899346013 4-107374182491' 2024-04-25T08:18:31.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:31.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836581 2024-04-25T08:18:31.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:31.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:18:31.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836581 2024-04-25T08:18:31.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:31.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836581 2024-04-25T08:18:31.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-04-25T08:18:31.235 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836581 2024-04-25T08:18:31.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:18:31.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836581 -lt 21474836581 2024-04-25T08:18:31.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:31.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673059 2024-04-25T08:18:31.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:31.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:18:31.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673059 2024-04-25T08:18:31.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:31.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673059 2024-04-25T08:18:31.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673059' 2024-04-25T08:18:31.674 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673059 2024-04-25T08:18:31.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:18:32.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673059 -lt 42949673059 2024-04-25T08:18:32.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:32.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509536 2024-04-25T08:18:32.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:32.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:18:32.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509536 2024-04-25T08:18:32.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:32.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509536 2024-04-25T08:18:32.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509536' 2024-04-25T08:18:32.137 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509536 2024-04-25T08:18:32.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:18:32.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509534 -lt 64424509536 2024-04-25T08:18:32.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:18:33.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:18:33.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:18:33.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509536 -lt 64424509536 2024-04-25T08:18:33.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:33.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346013 2024-04-25T08:18:33.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:33.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:18:33.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346013 2024-04-25T08:18:33.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:33.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346013 2024-04-25T08:18:33.985 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899346013 2024-04-25T08:18:33.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346013' 2024-04-25T08:18:33.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:18:34.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346014 -lt 85899346013 2024-04-25T08:18:34.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:34.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182491 2024-04-25T08:18:34.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:34.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:18:34.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182491 2024-04-25T08:18:34.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:34.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182491 2024-04-25T08:18:34.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182491' 2024-04-25T08:18:34.418 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182491 2024-04-25T08:18:34.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:18:34.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182491 -lt 107374182491 2024-04-25T08:18:34.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:18:34.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:18:34.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:18:34.841 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:18:34.841 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:34.841 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:18:34.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.1.asok 2024-04-25T08:18:34.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:18:34.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:18:34.949 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:18:34.950 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:34.951 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:34.951 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:18:34.951 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:18:34.951 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:34.951 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:18:34.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:18:35.336 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:18:35.336 INFO:tasks.workunit.client.0.smithi102.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-04-25T08:18:22.411751+0000 93'600 106:708 [1,4] 1 [1] 1 0'0 2024-04-25T08:11:58.482060+0000 0'0 2024-04-25T08:11:58.482060+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:18:35.336 INFO:tasks.workunit.client.0.smithi102.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:12:30.909558+0000 93'600 106:738 [1] 1 [1] 1 0'0 2024-04-25T08:11:47.100869+0000 0'0 2024-04-25T08:11:47.100869+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:41:57.489743+0000 0 0 2024-04-25T08:18:35.336 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T08:18:29.959384+0000 93'600 106:761 [1,2] 1 [1] 1 0'0 2024-04-25T08:11:41.048010+0000 0'0 2024-04-25T08:11:41.048010+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:18:35.337 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:18:35.337 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:18:35.337 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:18:35.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-04-25T08:18:36.069 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 3 size to 2 2024-04-25T08:18:36.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: sleep 2 2024-04-25T08:18:38.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:18:38.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:18:38.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:18:38.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:18:38.558 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:18:38.558 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:18:38.558 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:18:38.558 INFO:tasks.workunit.client.0.smithi102.stderr:4' 2024-04-25T08:18:38.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:18:38.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:38.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:18:38.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836585 2024-04-25T08:18:38.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836585 2024-04-25T08:18:38.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585' 2024-04-25T08:18:38.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:38.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:18:38.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673062 2024-04-25T08:18:38.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673062 2024-04-25T08:18:38.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673062' 2024-04-25T08:18:38.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:38.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:18:39.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509540 2024-04-25T08:18:39.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509540 2024-04-25T08:18:39.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673062 2-64424509540' 2024-04-25T08:18:39.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:39.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:18:39.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346017 2024-04-25T08:18:39.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346017 2024-04-25T08:18:39.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673062 2-64424509540 3-85899346017' 2024-04-25T08:18:39.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:39.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:18:39.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182494 2024-04-25T08:18:39.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182494 2024-04-25T08:18:39.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836585 1-42949673062 2-64424509540 3-85899346017 4-107374182494' 2024-04-25T08:18:39.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:39.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836585 2024-04-25T08:18:39.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:39.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:18:39.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836585 2024-04-25T08:18:39.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:39.325 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836585 2024-04-25T08:18:39.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836585 2024-04-25T08:18:39.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-04-25T08:18:39.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:18:39.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836585 -lt 21474836585 2024-04-25T08:18:39.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:39.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:39.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673062 2024-04-25T08:18:39.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:18:39.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673062 2024-04-25T08:18:39.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:39.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673062 2024-04-25T08:18:39.776 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673062 2024-04-25T08:18:39.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673062' 2024-04-25T08:18:39.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:18:40.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673062 -lt 42949673062 2024-04-25T08:18:40.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:40.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509540 2024-04-25T08:18:40.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:40.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:18:40.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509540 2024-04-25T08:18:40.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:40.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509540 2024-04-25T08:18:40.201 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509540 2024-04-25T08:18:40.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509540' 2024-04-25T08:18:40.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:18:40.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509538 -lt 64424509540 2024-04-25T08:18:40.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:18:41.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:18:41.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:18:42.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509540 -lt 64424509540 2024-04-25T08:18:42.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:42.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346017 2024-04-25T08:18:42.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:42.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:18:42.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346017 2024-04-25T08:18:42.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:42.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346017 2024-04-25T08:18:42.211 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899346017 2024-04-25T08:18:42.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346017' 2024-04-25T08:18:42.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:18:42.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346017 -lt 85899346017 2024-04-25T08:18:42.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:42.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182494 2024-04-25T08:18:42.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:42.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:18:42.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182494 2024-04-25T08:18:42.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:42.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182494 2024-04-25T08:18:42.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182494' 2024-04-25T08:18:42.662 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182494 2024-04-25T08:18:42.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:18:43.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182495 -lt 107374182494 2024-04-25T08:18:43.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:18:43.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:18:43.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:18:43.103 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:18:43.103 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:43.103 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:18:43.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.1.asok 2024-04-25T08:18:43.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:18:43.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:43.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:18:43.213 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:43.214 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:43.215 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:43.216 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:18:43.216 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:18:43.216 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:43.216 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:18:43.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:18:43.597 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:18:43.598 INFO:tasks.workunit.client.0.smithi102.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-04-25T08:18:22.411751+0000 93'600 110:715 [1,4] 1 [1] 1 0'0 2024-04-25T08:11:58.482060+0000 0'0 2024-04-25T08:11:58.482060+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:18:43.598 INFO:tasks.workunit.client.0.smithi102.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T08:18:37.773414+0000 93'600 110:746 [1,2] 1 [1] 1 0'0 2024-04-25T08:11:47.100869+0000 0'0 2024-04-25T08:11:47.100869+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:18:43.598 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T08:18:29.959384+0000 93'600 110:768 [1,2] 1 [1] 1 0'0 2024-04-25T08:11:41.048010+0000 0'0 2024-04-25T08:11:41.048010+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:18:43.598 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:18:43.598 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:18:43.598 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:18:43.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2024-04-25T08:18:43.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:18:43.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: PRIO=190 2024-04-25T08:18:43.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T08:18:43.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:18:43.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:18:43.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-04-25T08:18:43.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: ITEM=7.0 2024-04-25T08:18:43.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-04-25T08:18:43.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:18:43.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:18:43.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: PRIO=255 2024-04-25T08:18:43.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:203: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:18:43.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: seq 1 10 2024-04-25T08:18:43.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T08:18:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: ceph pg force-recovery 3.0 2024-04-25T08:18:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T08:18:44.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:214: TEST_recovery_priority: break 2024-04-25T08:18:44.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: sleep 2 2024-04-25T08:18:46.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:18:46.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:18:46.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:18:46.116 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:18:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:18:46.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.1.asok 2024-04-25T08:18:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:18:46.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:46.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:224: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:18:46.247 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:18:46.247 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:18:46.247 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:46.247 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:46.247 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:18:46.247 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:46.247 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 255, 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:18:46.248 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:18:46.249 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:18:46.250 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:46.250 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:46.250 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:18:46.250 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:18:46.250 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:46.250 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:18:46.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:18:46.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-04-25T08:18:46.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: PRIO=255 2024-04-25T08:18:46.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:18:46.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:231: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:18:46.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:18:46.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:18:46.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:18:46.693 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:18:46.693 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:18:46.693 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:18:46.693 INFO:tasks.workunit.client.0.smithi102.stderr:4' 2024-04-25T08:18:46.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:18:46.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:46.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:18:46.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836589 2024-04-25T08:18:46.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836589 2024-04-25T08:18:46.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836589' 2024-04-25T08:18:46.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:46.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:18:47.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673066 2024-04-25T08:18:47.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673066 2024-04-25T08:18:47.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836589 1-42949673066' 2024-04-25T08:18:47.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:47.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:18:47.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509543 2024-04-25T08:18:47.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509543 2024-04-25T08:18:47.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836589 1-42949673066 2-64424509543' 2024-04-25T08:18:47.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:47.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:18:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346021 2024-04-25T08:18:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346021 2024-04-25T08:18:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836589 1-42949673066 2-64424509543 3-85899346021' 2024-04-25T08:18:47.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:47.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:18:47.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182498 2024-04-25T08:18:47.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182498 2024-04-25T08:18:47.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836589 1-42949673066 2-64424509543 3-85899346021 4-107374182498' 2024-04-25T08:18:47.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:47.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836589 2024-04-25T08:18:47.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:47.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:18:47.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836589 2024-04-25T08:18:47.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:47.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836589 2024-04-25T08:18:47.520 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836589 2024-04-25T08:18:47.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836589' 2024-04-25T08:18:47.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:18:47.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836589 -lt 21474836589 2024-04-25T08:18:47.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:47.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673066 2024-04-25T08:18:47.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:47.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:18:47.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673066 2024-04-25T08:18:47.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:47.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673066 2024-04-25T08:18:47.927 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673066 2024-04-25T08:18:47.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673066' 2024-04-25T08:18:47.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:18:48.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673064 -lt 42949673066 2024-04-25T08:18:48.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:18:49.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:18:49.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:18:49.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673066 -lt 42949673066 2024-04-25T08:18:49.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:49.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509543 2024-04-25T08:18:49.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:49.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:18:49.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509543 2024-04-25T08:18:49.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:49.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509543 2024-04-25T08:18:49.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509543' 2024-04-25T08:18:49.773 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509543 2024-04-25T08:18:49.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:18:50.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509544 -lt 64424509543 2024-04-25T08:18:50.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:50.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346021 2024-04-25T08:18:50.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:50.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:18:50.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:50.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346021 2024-04-25T08:18:50.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346021 2024-04-25T08:18:50.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346021' 2024-04-25T08:18:50.244 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899346021 2024-04-25T08:18:50.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:18:50.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346021 -lt 85899346021 2024-04-25T08:18:50.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:50.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182498 2024-04-25T08:18:50.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:50.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:18:50.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182498 2024-04-25T08:18:50.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:50.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182498 2024-04-25T08:18:50.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182498' 2024-04-25T08:18:50.684 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182498 2024-04-25T08:18:50.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:18:51.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182499 -lt 107374182498 2024-04-25T08:18:51.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2024-04-25T08:18:51.516 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2024-04-25T08:18:51.516 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T08:18:51.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: sleep 2 2024-04-25T08:18:53.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:18:53.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:18:53.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:18:53.536 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:18:53.536 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:53.536 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:18:53.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.1.asok 2024-04-25T08:18:53.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:18:53.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:53.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:239: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:18:53.662 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:18:53.662 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:18:53.662 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:53.662 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:53.662 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:18:53.662 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:53.662 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:18:53.662 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:18:53.662 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:53.662 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:53.663 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:53.664 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:53.664 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:18:53.664 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:18:53.664 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:53.664 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:53.664 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:18:53.664 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:18:53.664 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:53.664 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:18:53.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:18:53.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2024-04-25T08:18:53.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: PRIO=190 2024-04-25T08:18:53.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T08:18:53.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:18:53.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:18:53.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: eval 'ITEM="3.0"' 2024-04-25T08:18:53.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: ITEM=3.0 2024-04-25T08:18:53.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: '[' 3.0 '!=' 3.0 ']' 2024-04-25T08:18:53.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:18:53.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:18:53.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: PRIO=255 2024-04-25T08:18:53.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:254: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:18:53.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:261: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2024-04-25T08:18:54.178 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2024-04-25T08:18:54.178 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T08:18:54.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:262: TEST_recovery_priority: sleep 5 2024-04-25T08:18:59.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:18:59.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:18:59.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:18:59.198 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:18:59.198 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:18:59.198 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:18:59.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.1.asok 2024-04-25T08:18:59.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:18:59.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:18:59.316 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:18:59.317 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:59.318 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:18:59.318 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:18:59.318 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:18:59.318 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:18:59.318 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:18:59.318 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:18:59.318 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:18:59.318 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:18:59.318 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:18:59.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:266: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:18:59.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:18:59.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:18:59.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:18:59.775 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:18:59.775 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:18:59.775 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:18:59.775 INFO:tasks.workunit.client.0.smithi102.stderr:4' 2024-04-25T08:18:59.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:18:59.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:59.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:18:59.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836593 2024-04-25T08:18:59.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836593 2024-04-25T08:18:59.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593' 2024-04-25T08:18:59.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:59.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:19:00.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673071 2024-04-25T08:19:00.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673071 2024-04-25T08:19:00.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673071' 2024-04-25T08:19:00.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:19:00.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:19:00.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509548 2024-04-25T08:19:00.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509548 2024-04-25T08:19:00.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509548' 2024-04-25T08:19:00.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:19:00.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:19:00.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346025 2024-04-25T08:19:00.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346025 2024-04-25T08:19:00.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509548 3-85899346025' 2024-04-25T08:19:00.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:19:00.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:19:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182503 2024-04-25T08:19:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182503 2024-04-25T08:19:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836593 1-42949673071 2-64424509548 3-85899346025 4-107374182503' 2024-04-25T08:19:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:19:00.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836593 2024-04-25T08:19:00.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:19:00.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:19:00.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836593 2024-04-25T08:19:00.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:19:00.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836593 2024-04-25T08:19:00.583 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836593 2024-04-25T08:19:00.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836593' 2024-04-25T08:19:00.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:19:00.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836591 -lt 21474836593 2024-04-25T08:19:00.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:19:01.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:19:01.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:19:02.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836593 -lt 21474836593 2024-04-25T08:19:02.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:19:02.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673071 2024-04-25T08:19:02.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:19:02.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:19:02.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673071 2024-04-25T08:19:02.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:19:02.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673071 2024-04-25T08:19:02.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673071' 2024-04-25T08:19:02.406 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673071 2024-04-25T08:19:02.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:19:02.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673071 -lt 42949673071 2024-04-25T08:19:02.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:19:02.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509548 2024-04-25T08:19:02.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:19:02.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:19:02.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509548 2024-04-25T08:19:02.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:19:02.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509548 2024-04-25T08:19:02.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509548' 2024-04-25T08:19:02.813 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509548 2024-04-25T08:19:02.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:19:03.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509548 -lt 64424509548 2024-04-25T08:19:03.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:19:03.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346025 2024-04-25T08:19:03.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:19:03.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:19:03.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346025 2024-04-25T08:19:03.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:19:03.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346025 2024-04-25T08:19:03.234 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899346025 2024-04-25T08:19:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346025' 2024-04-25T08:19:03.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:19:03.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346026 -lt 85899346025 2024-04-25T08:19:03.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:19:03.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182503 2024-04-25T08:19:03.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:19:03.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:19:03.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182503 2024-04-25T08:19:03.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:19:03.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182503 2024-04-25T08:19:03.732 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182503 2024-04-25T08:19:03.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182503' 2024-04-25T08:19:03.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:19:04.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182503 -lt 107374182503 2024-04-25T08:19:04.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T08:19:04.602 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2024-04-25T08:19:04.602 INFO:tasks.workunit.client.0.smithi102.stderr: 2024-04-25T08:19:04.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: sleep 2 2024-04-25T08:19:06.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:19:06.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:19:06.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:19:06.623 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:19:06.623 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:19:06.623 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:19:06.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.1.asok 2024-04-25T08:19:06.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:19:06.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:06.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:271: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:19:06.755 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:06.755 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:06.756 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:06.757 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:06.758 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:06.758 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:06.758 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:06.758 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:06.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:19:06.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-04-25T08:19:06.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: PRIO=190 2024-04-25T08:19:06.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T08:19:06.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:19:06.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:19:06.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-04-25T08:19:06.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: ITEM=7.0 2024-04-25T08:19:06.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-04-25T08:19:06.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:19:06.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:19:06.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: PRIO=255 2024-04-25T08:19:06.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:286: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:19:06.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: ceph osd unset noout 2024-04-25T08:19:07.543 INFO:tasks.workunit.client.0.smithi102.stderr:noout is unset 2024-04-25T08:19:07.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:294: TEST_recovery_priority: ceph osd unset norecover 2024-04-25T08:19:08.489 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is unset 2024-04-25T08:19:08.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:19:08.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:19:08.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:19:08.514 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:19:08.514 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:19:08.514 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:19:08.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.1.asok 2024-04-25T08:19:08.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations' 2024-04-25T08:19:08.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations' 2024-04-25T08:19:08.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:19:08.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:19:08.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:19:08.516 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:19:08.516 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:19:08.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:19:08.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:19:08.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:19:08.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:19:08.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:19:08.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:19:08.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:19:08.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:19:08.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:19:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:19:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:19:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:19:09.211 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:19:09.211 INFO:tasks.workunit.client.0.smithi102.stderr:4' 2024-04-25T08:19:09.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:19:09.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:19:09.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:19:09.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836597 2024-04-25T08:19:09.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836597 2024-04-25T08:19:09.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836597' 2024-04-25T08:19:09.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:19:09.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:19:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673074 2024-04-25T08:19:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673074 2024-04-25T08:19:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836597 1-42949673074' 2024-04-25T08:19:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:19:09.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:19:09.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509552 2024-04-25T08:19:09.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509552 2024-04-25T08:19:09.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836597 1-42949673074 2-64424509552' 2024-04-25T08:19:09.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:19:09.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:19:09.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899346029 2024-04-25T08:19:09.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899346029 2024-04-25T08:19:09.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836597 1-42949673074 2-64424509552 3-85899346029' 2024-04-25T08:19:09.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:19:09.838 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:19:09.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182506 2024-04-25T08:19:09.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182506 2024-04-25T08:19:09.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836597 1-42949673074 2-64424509552 3-85899346029 4-107374182506' 2024-04-25T08:19:09.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:19:09.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:19:09.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836597 2024-04-25T08:19:09.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:19:09.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836597 2024-04-25T08:19:09.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:19:10.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836597 2024-04-25T08:19:10.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836597' 2024-04-25T08:19:10.000 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836597 2024-04-25T08:19:10.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:19:10.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836595 -lt 21474836597 2024-04-25T08:19:10.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:19:11.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:19:11.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:19:11.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836597 -lt 21474836597 2024-04-25T08:19:11.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:19:11.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673074 2024-04-25T08:19:11.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:19:11.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:19:11.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673074 2024-04-25T08:19:11.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:19:11.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673074 2024-04-25T08:19:11.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673074' 2024-04-25T08:19:11.935 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673074 2024-04-25T08:19:11.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:19:12.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673075 -lt 42949673074 2024-04-25T08:19:12.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:19:12.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509552 2024-04-25T08:19:12.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:19:12.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:19:12.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509552 2024-04-25T08:19:12.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:19:12.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509552 2024-04-25T08:19:12.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509552' 2024-04-25T08:19:12.461 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509552 2024-04-25T08:19:12.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:19:12.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509552 -lt 64424509552 2024-04-25T08:19:12.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:19:12.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899346029 2024-04-25T08:19:12.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:19:12.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:19:12.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899346029 2024-04-25T08:19:12.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:19:12.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899346029 2024-04-25T08:19:12.902 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899346029 2024-04-25T08:19:12.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899346029' 2024-04-25T08:19:12.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:19:13.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899346030 -lt 85899346029 2024-04-25T08:19:13.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:19:13.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182506 2024-04-25T08:19:13.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:19:13.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:19:13.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182506 2024-04-25T08:19:13.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:19:13.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182506 2024-04-25T08:19:13.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182506' 2024-04-25T08:19:13.607 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182506 2024-04-25T08:19:13.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:19:14.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182507 -lt 107374182506 2024-04-25T08:19:14.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:19:14.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:14.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:14.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T08:19:14.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:14.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:14.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:14.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:14.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:14.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:14.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:15.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:15.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:15.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:15.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:15.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:19:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:19:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:19:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:15.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:15.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:15.816 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:15.816 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:15.816 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:15.816 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:15.817 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:15.818 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:15.819 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:15.819 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:15.819 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:15.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:15.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:15.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:15.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:15.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:15.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:15.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:15.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:15.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:16.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:16.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:16.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:16.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:16.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:16.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:16.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:16.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:16.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:16.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:16.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:16.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:16.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:17.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12096333 2024-04-25T08:19:17.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12096333 '!=' null 2024-04-25T08:19:17.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:17.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:17.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:17.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:17.725 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:17.725 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:17.725 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:17.725 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:17.725 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:17.726 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:17.727 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:17.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:17.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:17.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:17.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:17.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:17.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:17.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:17.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:17.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:18.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:18.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:18.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:18.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:18.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:18.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:18.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:18.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:18.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:18.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:18.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:18.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:18.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:19.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10171530 2024-04-25T08:19:19.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10171530 '!=' null 2024-04-25T08:19:19.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:19.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:19.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:19.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:19.521 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:19.521 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:19.521 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:19.521 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:19.521 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:19.522 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:19.523 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:19.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:19.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:19.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:19.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:19.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:19.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:19.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:19.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:19.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:20.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:20.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:20.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:20.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:20.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:20.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:20.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:20.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:20.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:20.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:20.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:20.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:20.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:21.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598278 2024-04-25T08:19:21.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598278 '!=' null 2024-04-25T08:19:21.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:21.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:21.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:21.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:21.309 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:21.310 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:21.310 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:21.310 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:21.310 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:21.310 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:21.310 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:21.311 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:21.311 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:21.311 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:21.311 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:21.311 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:21.311 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:21.312 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:21.312 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:21.312 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:21.312 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:21.312 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:21.312 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:21.313 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:21.313 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:21.313 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:21.313 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:21.313 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:21.313 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:21.314 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:21.314 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:21.314 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:21.314 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:21.314 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:21.314 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:21.314 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:21.315 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:21.315 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:21.315 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:21.315 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:21.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:21.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:21.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:21.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:21.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:21.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:21.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:21.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:21.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:21.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:21.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:21.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:21.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:22.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:22.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:22.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:22.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:22.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:22.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:22.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:22.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:22.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:23.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598278 2024-04-25T08:19:23.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598278 '!=' null 2024-04-25T08:19:23.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:23.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:23.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:23.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:23.175 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:23.176 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:23.176 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:23.176 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:23.176 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:23.176 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:23.176 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:23.176 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:23.176 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:23.176 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:23.176 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:23.176 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:23.177 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:23.178 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:23.178 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:23.178 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:23.178 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:23.178 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:23.178 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:23.178 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:23.178 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:23.178 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:23.178 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:23.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:23.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:23.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:23.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:23.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:23.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:23.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:23.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:23.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:23.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:23.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:23.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:24.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:24.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:24.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:24.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:24.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:24.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:24.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:24.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:24.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:24.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724816 2024-04-25T08:19:24.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724816 '!=' null 2024-04-25T08:19:24.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:24.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:24.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:24.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:25.094 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:25.095 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:25.096 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:25.096 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:25.096 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:25.096 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:25.096 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:25.096 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:25.096 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:25.096 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:25.096 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:25.096 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:25.096 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:25.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:25.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:25.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:25.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:25.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:25.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:25.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:25.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:25.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:25.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:25.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:25.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:25.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:26.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:26.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:26.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:26.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:26.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:26.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:26.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:26.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:26.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:26.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460509 2024-04-25T08:19:26.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460509 '!=' null 2024-04-25T08:19:26.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:26.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:26.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:26.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:27.039 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:27.039 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:27.039 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:27.039 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:27.039 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:27.039 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:27.039 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:27.039 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:27.039 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:27.040 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:27.041 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:27.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:27.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:27.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:27.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:27.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:27.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:27.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:27.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:27.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:27.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:27.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:27.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:27.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:28.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:28.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:28.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597511 2024-04-25T08:19:28.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597511 '!=' null 2024-04-25T08:19:28.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:28.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:28.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:28.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:29.015 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:29.015 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:29.016 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:29.017 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:29.018 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:29.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:29.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:29.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:29.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:29.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:29.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:29.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:29.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:29.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:29.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:29.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:29.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:29.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:30.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:30.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:30.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:30.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:30.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:30.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:30.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:30.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:30.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:30.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597732 2024-04-25T08:19:30.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597732 '!=' null 2024-04-25T08:19:30.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:30.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:30.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:30.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:30.803 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:30.804 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:30.805 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:30.806 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:30.806 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:30.806 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:30.806 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:30.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:30.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:30.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:30.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:30.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:30.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:30.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:30.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:30.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:31.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:31.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:31.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:31.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:32.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:32.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:32.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:32.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:32.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:32.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:32.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:32.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:32.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:32.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459421 2024-04-25T08:19:32.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459421 '!=' null 2024-04-25T08:19:32.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:32.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:32.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:32.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:32.662 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:32.663 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:32.664 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:32.665 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:32.665 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:32.665 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:32.665 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:32.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:32.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:32.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:32.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:32.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:32.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:32.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:32.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:32.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:33.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:33.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:33.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:33.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:33.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:33.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:33.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:33.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:33.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:33.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:33.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:33.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:33.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:34.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597094 2024-04-25T08:19:34.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597094 '!=' null 2024-04-25T08:19:34.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:34.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:34.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:34.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:34.352 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:34.352 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:34.352 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:34.352 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:34.352 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:34.352 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:34.352 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:34.352 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:34.352 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:34.352 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:34.353 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:34.354 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:34.354 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:34.354 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:34.354 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:34.354 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:34.354 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:34.354 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:34.354 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:34.354 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:34.354 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:34.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:34.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:34.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:34.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:34.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:34.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:34.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:34.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:34.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:34.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:34.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:34.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:34.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:35.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:35.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:35.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:35.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:35.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:35.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:35.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:35.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:35.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:36.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585861 2024-04-25T08:19:36.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585861 '!=' null 2024-04-25T08:19:36.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:36.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:36.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:36.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:36.207 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:36.207 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:36.207 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:36.207 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:36.207 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:36.207 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:36.208 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:36.209 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:36.210 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:36.210 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:36.210 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:36.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:36.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:36.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:36.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:36.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:36.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:36.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:36.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:36.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:36.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:36.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:36.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:36.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:37.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:37.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:37.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:37.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:37.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:37.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:37.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:37.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:37.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:37.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850699 2024-04-25T08:19:37.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850699 '!=' null 2024-04-25T08:19:37.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:37.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:37.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:37.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:38.032 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:38.032 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:38.032 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:38.032 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:38.032 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:38.032 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:38.033 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:38.034 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:38.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:38.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:38.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:38.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:38.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:38.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:38.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:38.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:38.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:38.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:38.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:38.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:38.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:39.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:39.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:39.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:39.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:39.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:39.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:39.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:39.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:39.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:39.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851182 2024-04-25T08:19:39.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851182 '!=' null 2024-04-25T08:19:39.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:39.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:39.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:39.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:40.016 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:40.016 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:40.016 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:40.016 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:40.016 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:40.016 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:40.016 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:40.016 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:40.016 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:40.016 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:40.017 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:40.018 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:40.018 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:40.018 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:40.018 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:40.018 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:40.018 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:40.018 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:40.018 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:40.018 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:40.018 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:40.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:40.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:40.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:40.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:40.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:40.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:40.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:40.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:40.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:40.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:40.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:40.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:40.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:41.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:41.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:41.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:41.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:41.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:41.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:41.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:41.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:41.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:41.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713099 2024-04-25T08:19:41.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713099 '!=' null 2024-04-25T08:19:41.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:41.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:41.640 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:41.640 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:41.741 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:41.741 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:41.742 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:41.743 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:41.744 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:41.744 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:41.744 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:41.744 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:41.744 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:41.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:41.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:41.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:41.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:41.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:41.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:41.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:41.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:41.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:42.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:42.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:42.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:42.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:42.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:42.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:42.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:42.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:42.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:42.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:42.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:42.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:42.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:43.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850823 2024-04-25T08:19:43.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850823 '!=' null 2024-04-25T08:19:43.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:43.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:43.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:43.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:43.687 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:43.687 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:43.687 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:43.687 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:43.687 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:43.687 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:43.688 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:43.689 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:43.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:43.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:44.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:44.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:44.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:44.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:44.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:44.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:44.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:44.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:44.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:44.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:44.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:44.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:44.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586380 2024-04-25T08:19:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586380 '!=' null 2024-04-25T08:19:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:45.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:45.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:45.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:45.631 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:45.631 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:45.631 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:45.631 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:45.631 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:45.631 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:45.631 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:45.631 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:45.632 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:45.633 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:45.634 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:45.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:45.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:45.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:45.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:45.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:45.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:45.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:45.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:45.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:46.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:46.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:46.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:46.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:46.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:46.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:46.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:46.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:46.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:46.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:47.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586380 2024-04-25T08:19:47.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586380 '!=' null 2024-04-25T08:19:47.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:47.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:47.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:47.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:47.459 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:47.460 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:47.461 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:47.461 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:47.461 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:47.461 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:47.461 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:47.461 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:47.461 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:47.461 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:47.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:47.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:47.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:47.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:47.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:47.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:47.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:47.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:47.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:48.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:48.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:48.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:48.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:48.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:48.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:48.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:48.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:48.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:48.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:48.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:48.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:48.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:49.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597573 2024-04-25T08:19:49.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597573 '!=' null 2024-04-25T08:19:49.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:49.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:49.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:49.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:49.344 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:49.345 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:49.346 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:49.346 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:49.346 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:49.346 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:49.346 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:49.346 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:49.346 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:49.346 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:49.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:49.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:49.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:49.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:49.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:49.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:49.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:49.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:49.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:49.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:49.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:49.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:49.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:50.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:50.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:50.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:50.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:50.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:50.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:50.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:50.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:50.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:51.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597573 2024-04-25T08:19:51.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597573 '!=' null 2024-04-25T08:19:51.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:51.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:51.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:51.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:51.247 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:51.248 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:51.249 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:51.250 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:51.250 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:51.250 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:51.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:51.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:51.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:51.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:51.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:51.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:51.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:51.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:51.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:51.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:51.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:51.918 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:51.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:52.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:52.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:52.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:52.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:52.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:52.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:52.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:52.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:52.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:53.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459326 2024-04-25T08:19:53.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459326 '!=' null 2024-04-25T08:19:53.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:53.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:53.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:53.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:53.423 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:53.423 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:53.423 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:53.423 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:53.423 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:53.423 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:53.423 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:53.423 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:53.423 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:53.424 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:53.425 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:53.425 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:53.425 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:53.425 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:53.425 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:53.425 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:53.425 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:53.425 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:53.425 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:53.425 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:53.425 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:53.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:53.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:53.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:53.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:53.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:53.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:53.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:53.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:53.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:54.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:54.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:54.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:54.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:54.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:54.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:54.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:54.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:54.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:54.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:54.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:54.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:54.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459464 2024-04-25T08:19:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459464 '!=' null 2024-04-25T08:19:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:55.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:55.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:55.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:55.394 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:55.394 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:55.394 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:55.394 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:55.394 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:55.394 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:55.394 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:55.394 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:55.394 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:55.394 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:55.395 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:55.396 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:55.396 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:55.396 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:55.396 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:55.396 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:55.396 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:55.396 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:55.396 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:55.396 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:55.396 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:55.396 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:55.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:55.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:55.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:55.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:55.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:55.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:55.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:55.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:55.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:55.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:55.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:55.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:55.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:56.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:56.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:56.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:56.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:56.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:56.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:56.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:56.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:56.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:57.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459464 2024-04-25T08:19:57.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459464 '!=' null 2024-04-25T08:19:57.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:57.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:57.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:57.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:57.136 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:57.136 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:57.136 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:57.136 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:57.136 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:57.137 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:57.138 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:57.139 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:57.139 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:57.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:57.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:57.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:57.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:57.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:57.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:57.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:57.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:57.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:57.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:57.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:57.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:57.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:19:58.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:19:58.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:19:58.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:19:58.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:19:58.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:19:58.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:19:58.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:19:58.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:19:58.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:19:58.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724030 2024-04-25T08:19:58.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724030 '!=' null 2024-04-25T08:19:58.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:19:58.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:58.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:19:58.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:19:59.047 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:19:59.047 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:19:59.047 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:59.047 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:59.047 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:59.048 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:19:59.049 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:19:59.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:19:59.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:19:59.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:19:59.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:19:59.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:19:59.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:19:59.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:19:59.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:19:59.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:19:59.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:19:59.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:19:59.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:19:59.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:00.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:00.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:00.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:00.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:00.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:00.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:00.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:00.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:00.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:00.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723780 2024-04-25T08:20:00.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723780 '!=' null 2024-04-25T08:20:00.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:00.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:00.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:00.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:00.888 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:00.888 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:00.889 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:00.890 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:00.890 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:00.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:01.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:01.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:01.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:01.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:01.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:01.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:01.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:01.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:01.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:01.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:01.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:01.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:01.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:01.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:01.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:01.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:01.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:01.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:01.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:02.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585624 2024-04-25T08:20:02.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585624 '!=' null 2024-04-25T08:20:02.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:02.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:02.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:02.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:02.664 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:02.664 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:02.664 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:02.664 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:02.664 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:02.664 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:02.664 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:02.664 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:02.664 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:02.664 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:02.664 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:02.665 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:02.666 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:02.666 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:02.666 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:02.666 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:02.666 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:02.666 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:02.666 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:02.666 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:02.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:02.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:02.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:02.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:02.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:02.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:02.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:02.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:02.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:03.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:03.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:03.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:03.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:04.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:04.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:04.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:04.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:04.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:04.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:04.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:04.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:04.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:04.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723613 2024-04-25T08:20:04.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723613 '!=' null 2024-04-25T08:20:04.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:04.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:04.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:04.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:04.829 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:04.829 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:04.829 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:04.829 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:04.829 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:04.829 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:04.829 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:04.829 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:04.829 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:04.829 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:04.830 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:04.831 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:04.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:04.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:04.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:04.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:04.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:04.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:04.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:04.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:04.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:05.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:05.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:05.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:05.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:05.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:05.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:05.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:05.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:05.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:05.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:05.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:05.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:05.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:06.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585849 2024-04-25T08:20:06.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585849 '!=' null 2024-04-25T08:20:06.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:06.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:06.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:06.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:06.710 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:06.711 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:06.712 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:06.712 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:06.712 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:06.712 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:06.712 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:06.712 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:06.712 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:06.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:06.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:06.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:06.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:06.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:06.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:06.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:06.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:06.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:07.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:07.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:07.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:07.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:07.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:07.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:07.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:07.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:07.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:07.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:07.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:07.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:07.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:08.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723897 2024-04-25T08:20:08.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723897 '!=' null 2024-04-25T08:20:08.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:08.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:08.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:08.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:08.598 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:08.598 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:08.598 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:08.598 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:08.598 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:08.598 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:08.598 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:08.598 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:08.599 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:08.600 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:08.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:08.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:08.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:08.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:08.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:08.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:08.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:08.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:08.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:09.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:09.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:09.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:09.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:09.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:09.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:09.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:09.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:09.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:09.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:09.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:09.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:09.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:10.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723980 2024-04-25T08:20:10.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723980 '!=' null 2024-04-25T08:20:10.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:10.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:10.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:10.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:10.494 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:10.494 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:10.494 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:10.494 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:10.494 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:10.494 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:10.494 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:10.494 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:10.495 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:10.496 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:10.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:10.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:10.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:10.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:10.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:10.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:10.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:10.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:10.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:11.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:11.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:11.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:11.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:11.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:11.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:11.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:11.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:12.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586284 2024-04-25T08:20:12.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586284 '!=' null 2024-04-25T08:20:12.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:12.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:12.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:12.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:12.356 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:12.357 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:12.358 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:12.359 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:12.359 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:12.359 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:12.359 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:12.359 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:12.359 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:12.359 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:12.359 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:12.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:12.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:12.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:12.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:12.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:12.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:12.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:12.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:12.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:12.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:12.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:12.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:12.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:13.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:13.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:13.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:13.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:13.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:13.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:13.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:13.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:13.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:14.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724238 2024-04-25T08:20:14.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724238 '!=' null 2024-04-25T08:20:14.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:14.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:14.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:14.064 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:14.170 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:14.170 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:14.171 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:14.172 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:14.173 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:14.173 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:14.173 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:14.173 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:14.173 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:14.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:14.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:14.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:14.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:14.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:14.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:14.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:14.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:14.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:14.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:14.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:14.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:15.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:15.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:15.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:15.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:15.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:15.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:15.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:15.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:15.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:15.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460614 2024-04-25T08:20:15.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460614 '!=' null 2024-04-25T08:20:15.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:15.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:15.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:15.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:15.937 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:15.937 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:15.938 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:15.938 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:15.938 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:15.938 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:15.938 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:15.938 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:15.938 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:15.939 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:15.939 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:15.939 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:15.939 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:15.939 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:15.939 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:15.939 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:15.939 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:15.939 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:15.939 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:15.940 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:15.941 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:15.941 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:15.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:16.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:16.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:16.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:16.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:16.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:16.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:16.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:16.054 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:16.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:16.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:16.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:16.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:16.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:16.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:16.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:16.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:16.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:16.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:16.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:16.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:16.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:17.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598015 2024-04-25T08:20:17.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598015 '!=' null 2024-04-25T08:20:17.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:17.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:17.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:17.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:17.752 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:17.752 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:17.752 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:17.752 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:17.752 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:17.753 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:17.754 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:17.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:17.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:17.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:17.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:17.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:17.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:17.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:17.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:17.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:18.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:18.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:18.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:18.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:18.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:18.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:18.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:18.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:18.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:18.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:18.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:18.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:18.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:19.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598215 2024-04-25T08:20:19.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598215 '!=' null 2024-04-25T08:20:19.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:19.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:19.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:19.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:19.793 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:19.793 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:19.793 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:19.793 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:19.793 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:19.793 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:19.794 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:19.795 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:19.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:19.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:19.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:19.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:19.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:19.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:19.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:19.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:19.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:20.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:20.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:20.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:20.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:20.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:20.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:20.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:20.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:20.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:20.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:20.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:20.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:20.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:21.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460286 2024-04-25T08:20:21.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460286 '!=' null 2024-04-25T08:20:21.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:21.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:21.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:21.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:21.629 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:21.629 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:21.629 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:21.629 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:21.629 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:21.629 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:21.629 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:21.630 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:21.631 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:21.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:21.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:21.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:21.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:21.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:21.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:21.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:21.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:21.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:22.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:22.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:22.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:22.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:22.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:22.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:22.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:22.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:22.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:22.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:22.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:22.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:22.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:23.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598207 2024-04-25T08:20:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598207 '!=' null 2024-04-25T08:20:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:23.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:23.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:23.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:23.672 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:23.673 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:23.674 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:23.674 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:23.674 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:23.674 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:23.674 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:23.674 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:23.674 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:23.674 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:23.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:23.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:23.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:23.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:23.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:23.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:23.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:23.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:23.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:24.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:24.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:24.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:24.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:24.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:24.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:24.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:24.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:24.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:24.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:24.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:24.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:24.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:25.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460913 2024-04-25T08:20:25.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460913 '!=' null 2024-04-25T08:20:25.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:25.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:25.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:25.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:25.545 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:25.545 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:25.545 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:25.545 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:25.545 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:25.545 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:25.545 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:25.545 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:25.545 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:25.546 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:25.547 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:25.547 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:25.547 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:25.547 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:25.547 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:25.547 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:25.547 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:25.547 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:25.547 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:25.547 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:25.547 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:25.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:25.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:25.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:25.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:25.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:25.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:25.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:25.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:25.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:26.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:26.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:26.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:26.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:26.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:26.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:26.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:26.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:26.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:26.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:26.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:26.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:26.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:27.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724838 2024-04-25T08:20:27.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724838 '!=' null 2024-04-25T08:20:27.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:27.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:27.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:27.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:27.320 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:27.320 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:27.320 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:27.321 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:27.321 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:27.321 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:27.321 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:27.321 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:27.322 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:27.322 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:27.322 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:27.322 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:27.322 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:27.322 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:27.323 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:27.323 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:27.323 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:27.323 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:27.323 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:27.323 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:27.324 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:27.324 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:27.324 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:27.324 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:27.324 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:27.324 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:27.325 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:27.325 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:27.325 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:27.325 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:27.325 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:27.325 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:27.326 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:27.326 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:27.326 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:27.326 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:27.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:27.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:27.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:27.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:27.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:27.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:27.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:27.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:27.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:27.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:27.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:27.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:27.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:28.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:28.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:28.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:28.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:28.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:28.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:28.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:28.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:28.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:29.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724838 2024-04-25T08:20:29.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724838 '!=' null 2024-04-25T08:20:29.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:29.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:29.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:29.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:29.369 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:29.369 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:29.369 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:29.369 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:29.369 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:29.369 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:29.369 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:29.369 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:29.369 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:29.369 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:29.369 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:29.370 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:29.373 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:29.373 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:29.373 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:29.373 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:29.373 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:29.373 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:29.374 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:29.374 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:29.374 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:29.374 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:29.374 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:29.374 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:29.374 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:29.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:29.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:29.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:29.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:29.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:29.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:29.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:29.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:29.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:29.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:29.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:29.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:29.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:30.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:30.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:30.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:30.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:30.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:30.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:30.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:30.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:30.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:30.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724925 2024-04-25T08:20:30.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724925 '!=' null 2024-04-25T08:20:30.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:30.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:30.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:30.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:31.102 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:31.102 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:31.102 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:31.102 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:31.102 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:31.102 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:31.102 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:31.102 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:31.102 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:31.102 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:31.103 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:31.104 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:31.104 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:31.104 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:31.104 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:31.104 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:31.104 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:31.104 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:31.104 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:31.104 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:31.104 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:31.104 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:31.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:31.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:31.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:31.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:31.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:31.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:31.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:31.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:31.217 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:31.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:31.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:31.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:31.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:32.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:32.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:32.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:32.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:32.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:32.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:32.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:32.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:32.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:32.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23587498 2024-04-25T08:20:32.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23587498 '!=' null 2024-04-25T08:20:32.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:32.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:32.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:33.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:33.099 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:33.099 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:33.099 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:33.099 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:33.099 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:33.099 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:33.099 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:33.099 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:33.099 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:33.099 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:33.100 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:33.101 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:33.101 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:33.101 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:33.101 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:33.101 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:33.101 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:33.101 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:33.101 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:33.101 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:33.101 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:33.101 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:33.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:33.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:33.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:33.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:33.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:33.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:33.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:33.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:33.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:33.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:33.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:33.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:33.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:34.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:34.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:34.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:34.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:34.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:34.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:34.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:34.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:34.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:34.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725311 2024-04-25T08:20:34.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725311 '!=' null 2024-04-25T08:20:34.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:34.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:34.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:34.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:35.007 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:35.008 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:35.009 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:35.010 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:35.010 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:35.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:35.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:35.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:35.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:35.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:35.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:35.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:35.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:35.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:35.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:35.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:35.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:35.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:36.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:36.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:36.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:36.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:36.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:36.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:36.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:36.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:36.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713714 2024-04-25T08:20:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713714 '!=' null 2024-04-25T08:20:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:36.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:36.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:36.904 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:36.904 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:36.904 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:36.904 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:36.904 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:36.904 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:36.904 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:36.904 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:36.905 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:36.906 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:36.907 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:36.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:37.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:37.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:37.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:37.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:37.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:37.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:37.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:37.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:37.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:37.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:37.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:37.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:38.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:38.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:38.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:38.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:38.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:38.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:38.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:38.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:38.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:38.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850927 2024-04-25T08:20:38.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850927 '!=' null 2024-04-25T08:20:38.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:38.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:38.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:38.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:38.792 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:38.792 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:38.792 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:38.793 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:38.794 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:38.795 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:38.795 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:38.795 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:38.795 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:38.795 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:38.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:38.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:38.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:38.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:38.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:38.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:38.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:38.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:38.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:39.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:39.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:39.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:39.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:39.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:39.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:39.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:39.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:40.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850677 2024-04-25T08:20:40.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850677 '!=' null 2024-04-25T08:20:40.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:40.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:40.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:40.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:40.575 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:40.575 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:40.575 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:40.576 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:40.577 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:40.578 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:40.578 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:40.578 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:40.578 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:40.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:40.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:40.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:40.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:40.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:40.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:40.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:40.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:40.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:41.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:41.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:41.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:41.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:41.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:41.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:41.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:41.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:41.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:41.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:41.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:41.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:41.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:42.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712738 2024-04-25T08:20:42.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712738 '!=' null 2024-04-25T08:20:42.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:42.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:42.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:42.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:42.347 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:42.347 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:42.348 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:42.349 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:42.350 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:42.350 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:42.350 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:42.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:42.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:42.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:42.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:42.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:42.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:42.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:42.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:42.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:42.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:42.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:42.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:42.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:43.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:43.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:43.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:43.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:43.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:43.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:43.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:43.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:43.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:44.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850494 2024-04-25T08:20:44.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850494 '!=' null 2024-04-25T08:20:44.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:44.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:44.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:44.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:44.212 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:44.213 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:44.214 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:44.215 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:44.215 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:44.215 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:44.215 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:44.215 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:44.215 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:44.215 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:44.215 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:44.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:44.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:44.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:44.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:44.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:44.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:44.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:44.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:44.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:44.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:44.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:44.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:44.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:45.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:45.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:45.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:45.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:45.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:45.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:45.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:45.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:45.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:45.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712172 2024-04-25T08:20:45.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712172 '!=' null 2024-04-25T08:20:45.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:45.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:45.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:45.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:46.065 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:46.065 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:46.065 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:46.065 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:46.065 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:46.065 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:46.065 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:46.065 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:46.065 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:46.065 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:46.066 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:46.067 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:46.067 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:46.067 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:46.067 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:46.067 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:46.067 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:46.067 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:46.067 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:46.067 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:46.067 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:46.067 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:46.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:46.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:46.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:46.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:46.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:46.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:46.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:46.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:47.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:47.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:47.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:47.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:47.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:47.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:47.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:47.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:47.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:47.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724001 2024-04-25T08:20:47.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724001 '!=' null 2024-04-25T08:20:47.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:47.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:47.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:47.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:47.871 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:47.871 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:47.871 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:47.871 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:47.871 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:47.872 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:47.873 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:47.874 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:47.874 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:47.874 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:47.874 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:47.874 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:47.874 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:47.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:47.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:47.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:47.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:47.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:47.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:47.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:47.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:47.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:48.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:48.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:48.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:48.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:49.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:49.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:49.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:49.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:49.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:49.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:49.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:49.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:49.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:49.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723919 2024-04-25T08:20:49.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723919 '!=' null 2024-04-25T08:20:49.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:49.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:49.836 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:49.836 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:49.943 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:49.943 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:49.943 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:49.943 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:49.943 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:49.943 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:49.943 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:49.943 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:49.943 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:49.943 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:49.944 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:49.945 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:49.945 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:49.945 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:49.945 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:49.945 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:49.945 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:49.945 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:49.945 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:49.945 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:49.945 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:49.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:50.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:50.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:50.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:50.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:50.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:50.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:50.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:50.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:50.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:50.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:50.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:50.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:50.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:50.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:50.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:50.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:50.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:50.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:50.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:50.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:50.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:51.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585811 2024-04-25T08:20:51.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585811 '!=' null 2024-04-25T08:20:51.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:51.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:51.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:51.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:51.826 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:51.826 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:51.826 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:51.826 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:51.826 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:51.826 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:51.827 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:51.828 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:51.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:51.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:51.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:51.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:51.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:51.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:51.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:51.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:51.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:52.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:52.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:52.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:52.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:52.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:52.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:52.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:52.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:52.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:52.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:52.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:52.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:52.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:53.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585811 2024-04-25T08:20:53.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585811 '!=' null 2024-04-25T08:20:53.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:53.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:53.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:53.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:53.650 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:53.651 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:53.652 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:53.653 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:53.653 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:53.653 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:53.653 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:53.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:53.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:53.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:53.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:53.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:53.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:53.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:53.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:53.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:54.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:54.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:54.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:54.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:54.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:54.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:54.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:54.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:54.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:54.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:54.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:54.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:54.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23584741 2024-04-25T08:20:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23584741 '!=' null 2024-04-25T08:20:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:55.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:55.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:55.609 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:55.609 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:55.610 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:55.611 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:55.612 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:55.612 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:55.612 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:55.612 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:55.612 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:55.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:55.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:55.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:55.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:55.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:55.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:55.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:55.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:55.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:56.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:56.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:56.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:56.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:56.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:56.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:56.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:56.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:56.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:56.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:56.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:56.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:56.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:57.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723558 2024-04-25T08:20:57.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723558 '!=' null 2024-04-25T08:20:57.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:57.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:57.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:57.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:57.497 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:57.497 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:57.498 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:57.499 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:57.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:57.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:57.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:57.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:57.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:57.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:57.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:57.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:57.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:58.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:58.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:58.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:58.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:20:58.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:20:58.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:20:58.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:20:58.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:20:58.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:20:58.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:20:58.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:20:58.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:20:58.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:20:59.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723670 2024-04-25T08:20:59.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723670 '!=' null 2024-04-25T08:20:59.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:20:59.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:59.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:20:59.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:20:59.385 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:20:59.385 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:20:59.385 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:59.385 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:59.385 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:20:59.385 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:59.385 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:20:59.385 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:59.386 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "7.0", 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 255, 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:20:59.387 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:20:59.388 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:20:59.388 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:20:59.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:20:59.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:20:59.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:20:59.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:20:59.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:20:59.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:20:59.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:20:59.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:20:59.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:20:59.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:20:59.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:20:59.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:20:59.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:00.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:21:00.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:21:00.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:00.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:00.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:00.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:00.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:00.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:00.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:01.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723670 2024-04-25T08:21:01.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723670 '!=' null 2024-04-25T08:21:01.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:01.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:01.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:01.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:01.963 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:01.963 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:01.963 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:01.963 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:01.964 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:01.965 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:01.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:02.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:02.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:02.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:02.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:02.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:02.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:02.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:02.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:02.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:21:02.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:02.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:02.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:03.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:21:03.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:21:03.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:03.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:03.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:03.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:03.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:03.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:03.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:03.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11793525 2024-04-25T08:21:03.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11793525 '!=' null 2024-04-25T08:21:03.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:03.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:03.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:03.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:04.091 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:04.091 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:04.091 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:04.091 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:04.091 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:04.091 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:04.091 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:04.091 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:04.091 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:04.091 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:04.091 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:04.092 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:04.093 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:04.093 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:04.093 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:04.093 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:04.093 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:04.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:04.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:04.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:04.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:04.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:04.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:04.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:04.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:04.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:04.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:21:04.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:04.623 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:04.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:05.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:21:05.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:21:05.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:05.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:05.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:05.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:05.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:05.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:05.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:06.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206697 2024-04-25T08:21:06.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206697 '!=' null 2024-04-25T08:21:06.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:06.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:06.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:06.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:06.142 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:06.143 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:06.143 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:06.143 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:06.143 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:06.143 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:06.144 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:06.144 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:06.144 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:06.144 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:06.144 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:06.144 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:06.145 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:06.145 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:06.145 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:06.145 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:06.145 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:06.145 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:06.146 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:06.146 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:06.146 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:06.146 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:06.146 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:06.146 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:06.147 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:06.147 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:06.147 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:06.147 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:06.147 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:06.147 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:06.148 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:06.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:06.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:06.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:06.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:06.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:06.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:06.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:06.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:06.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:06.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:06.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:07.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:07.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-04-25T08:21:07.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:21:07.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-04-25T08:21:07.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:07.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:07.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:07.321 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:07.322 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:07.323 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:07.323 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:07.323 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:07.323 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:07.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:07.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:07.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:07.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:07.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:07.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:07.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:07.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:07.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:07.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:07.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:07.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:07.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:08.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:08.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:08.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:08.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:08.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:08.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:08.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:08.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:08.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:09.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206104 2024-04-25T08:21:09.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206104 '!=' null 2024-04-25T08:21:09.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:09.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:09.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:09.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:09.243 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:09.244 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:09.245 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:09.246 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:09.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:09.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:09.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:09.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:09.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:09.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:09.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:09.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:09.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:09.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:09.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:09.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:09.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:10.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:10.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:10.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:10.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:10.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:10.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:10.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:10.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:10.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:11.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12635157 2024-04-25T08:21:11.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12635157 '!=' null 2024-04-25T08:21:11.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:11.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:11.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:11.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:11.136 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:11.136 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:11.136 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:11.136 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:11.136 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:11.136 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:11.136 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:11.137 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:11.138 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:11.138 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:11.138 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:11.138 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:11.138 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:11.138 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:11.138 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:11.138 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:11.138 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:11.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:11.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:11.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:11.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:11.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:11.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:11.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:11.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:11.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:11.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:11.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:11.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:12.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:12.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:12.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:12.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:12.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:12.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:12.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:12.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:12.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:12.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19048812 2024-04-25T08:21:12.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19048812 '!=' null 2024-04-25T08:21:12.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:12.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:12.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:12.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:12.971 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:12.972 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:12.973 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:12.973 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:12.973 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:12.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:13.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:13.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:13.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:13.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:13.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:13.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:13.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:13.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:13.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:13.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:13.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:13.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:14.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:14.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:14.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:14.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:14.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:14.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:14.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:14.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:14.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:14.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597686 2024-04-25T08:21:14.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597686 '!=' null 2024-04-25T08:21:14.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:14.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:14.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:14.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:15.089 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:15.089 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:15.090 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:15.090 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:15.090 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:15.090 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:15.090 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:15.091 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:15.091 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:15.091 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:15.091 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:15.091 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:15.091 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:15.092 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:15.092 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:15.092 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:15.092 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:15.092 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:15.093 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:15.093 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:15.093 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:15.093 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:15.093 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:15.093 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:15.093 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:15.094 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:15.094 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:15.094 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:15.094 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:15.094 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:15.094 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:15.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:15.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:15.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:15.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:15.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:15.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:15.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:15.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:15.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:15.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:15.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:15.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:16.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:16.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:16.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:16.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:16.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:16.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:16.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:16.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:16.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:16.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333669 2024-04-25T08:21:16.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333669 '!=' null 2024-04-25T08:21:16.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:16.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:16.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:16.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:16.879 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:16.879 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:16.879 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:16.879 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:16.879 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:16.879 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:16.879 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:16.880 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:16.881 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:16.881 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:16.881 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:16.881 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:16.881 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:16.881 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:16.881 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:16.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:16.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:16.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:16.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:16.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:16.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:16.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:16.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:16.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:17.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:17.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:17.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:17.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:18.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:18.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:18.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:18.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:18.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:18.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:18.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:18.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:18.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:18.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724305 2024-04-25T08:21:18.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724305 '!=' null 2024-04-25T08:21:18.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:18.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:18.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:18.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:18.756 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:18.756 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:18.756 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:18.756 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:18.756 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:18.756 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:18.756 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:18.756 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:18.756 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:18.756 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:18.756 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:18.757 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:18.758 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:18.758 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:18.758 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:18.758 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:18.758 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:18.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:18.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:18.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:18.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:18.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:18.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:18.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:18.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:18.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:19.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:19.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:19.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:19.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:20.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:20.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:20.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:20.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:20.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:20.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:20.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:20.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:20.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:20.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724569 2024-04-25T08:21:20.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724569 '!=' null 2024-04-25T08:21:20.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:20.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:20.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:20.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:20.845 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:20.845 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:20.845 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:20.845 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:20.846 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:20.847 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:20.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:20.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:20.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:20.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:20.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:20.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:20.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:20.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:20.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:21.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:21.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:21.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:21.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:22.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:22.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:22.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:22.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:22.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:22.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:22.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:22.045 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:22.045 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:22.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713533 2024-04-25T08:21:22.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713533 '!=' null 2024-04-25T08:21:22.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:22.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:22.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:22.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:22.850 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:22.850 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:22.851 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:22.852 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:22.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:22.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:22.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:22.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:22.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:22.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:22.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:22.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:22.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:23.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:23.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:23.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:23.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:24.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:24.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:24.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:24.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:24.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:24.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:24.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:24.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:24.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:24.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724595 2024-04-25T08:21:24.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724595 '!=' null 2024-04-25T08:21:24.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:24.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:24.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:24.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:24.777 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:24.777 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:24.777 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:24.777 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:24.778 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:24.779 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:24.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:24.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:24.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:24.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:24.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:24.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:24.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:24.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:24.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:25.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:25.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:25.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:25.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:25.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:25.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:25.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:26.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586341 2024-04-25T08:21:26.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586341 '!=' null 2024-04-25T08:21:26.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:26.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:26.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:26.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:26.620 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:26.621 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:26.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:26.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:26.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:26.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:26.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:26.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:26.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:26.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:26.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:27.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:27.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:27.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:27.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:27.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:27.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:27.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:27.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:27.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:27.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:27.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:27.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:27.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:28.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850416 2024-04-25T08:21:28.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850416 '!=' null 2024-04-25T08:21:28.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:28.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:28.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:28.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:28.608 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:28.608 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:28.608 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:28.608 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:28.608 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:28.608 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:28.608 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:28.608 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:28.608 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:28.609 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:28.610 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:28.610 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:28.610 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:28.610 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:28.610 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:28.610 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:28.610 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:28.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:28.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:28.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:28.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:28.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:28.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:28.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:28.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:28.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:29.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:29.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:29.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:29.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:29.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:29.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:29.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:29.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:29.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:29.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:29.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:29.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:29.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:30.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850511 2024-04-25T08:21:30.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850511 '!=' null 2024-04-25T08:21:30.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:30.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:30.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:30.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:30.337 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:30.337 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:30.338 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:30.339 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:30.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:30.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:30.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:30.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:30.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:30.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:30.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:30.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:30.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:30.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:30.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:30.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:31.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:31.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:31.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:31.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:31.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:31.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:31.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:31.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:31.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:32.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838649 2024-04-25T08:21:32.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838649 '!=' null 2024-04-25T08:21:32.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:32.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:32.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:32.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:32.187 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:32.187 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:32.188 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:32.189 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:32.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:32.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:32.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:32.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:32.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:32.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:32.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:32.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:32.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:32.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:32.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:32.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:33.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:33.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:33.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:33.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:33.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:33.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:33.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:33.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:33.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:33.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850136 2024-04-25T08:21:33.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850136 '!=' null 2024-04-25T08:21:33.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:33.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:33.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:33.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:34.023 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:34.024 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:34.025 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:34.025 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:34.025 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:34.025 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:34.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:34.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:34.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:34.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:34.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:34.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:34.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:34.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:34.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:34.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:34.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:34.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:34.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:35.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:35.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:35.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:35.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:35.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:35.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:35.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:35.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:35.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:35.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585887 2024-04-25T08:21:35.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585887 '!=' null 2024-04-25T08:21:35.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:35.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:35.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:35.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:35.863 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:35.864 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:35.865 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:35.866 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:35.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:35.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:35.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:35.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:35.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:35.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:35.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:35.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:35.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:36.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:36.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:36.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:36.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:36.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:36.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:36.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:36.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:36.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:36.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:36.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:36.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:36.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:37.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723984 2024-04-25T08:21:37.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723984 '!=' null 2024-04-25T08:21:37.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:37.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:37.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:37.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:37.715 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:37.716 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:37.717 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:37.718 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:37.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:37.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:37.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:37.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:37.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:37.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:37.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:37.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:37.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:38.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:38.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:38.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:38.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:38.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:38.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:38.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:38.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:38.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:38.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:38.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:38.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:38.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724355 2024-04-25T08:21:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724355 '!=' null 2024-04-25T08:21:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:39.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:39.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:39.479 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:39.480 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:39.481 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:39.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:39.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:39.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:39.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:39.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:39.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:39.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:39.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:40.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:40.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:40.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:40.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:40.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:40.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:40.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:40.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:40.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:40.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:40.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:40.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:40.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:41.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713588 2024-04-25T08:21:41.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713588 '!=' null 2024-04-25T08:21:41.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:41.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:41.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:41.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:41.266 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:41.266 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:41.266 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:41.266 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:41.267 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:41.268 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:41.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:41.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:41.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:41.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:41.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:41.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:41.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:41.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:41.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:41.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:41.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:41.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:41.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:42.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:42.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:42.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:42.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:42.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:42.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:42.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:42.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:42.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713588 2024-04-25T08:21:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713588 '!=' null 2024-04-25T08:21:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:42.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:42.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:42.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:43.017 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:43.018 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:43.019 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:43.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:43.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:43.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:43.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:43.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:43.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:43.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:43.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:43.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:43.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:43.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:43.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:43.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:44.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:44.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:44.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:44.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:44.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:44.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:44.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:44.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:44.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:44.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724768 2024-04-25T08:21:44.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724768 '!=' null 2024-04-25T08:21:44.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:44.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:44.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:44.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:44.967 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:44.968 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:44.969 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:44.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:45.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:45.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:45.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:45.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:45.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:45.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:45.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:45.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:45.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:45.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:45.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:45.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:46.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:46.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:46.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:46.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:46.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:46.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:46.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:46.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:46.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:46.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23587137 2024-04-25T08:21:46.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23587137 '!=' null 2024-04-25T08:21:46.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:46.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:46.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:46.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:46.988 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:46.988 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:46.988 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:46.989 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:46.990 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:46.990 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:46.990 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:46.990 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:46.990 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:46.990 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:46.990 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:46.990 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:46.990 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:46.990 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:46.990 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:47.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:47.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:47.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:47.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:47.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:47.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:47.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:47.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:47.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:47.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:47.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:47.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:47.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:48.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:48.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:48.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:48.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:48.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:48.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:48.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:48.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:48.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:48.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851002 2024-04-25T08:21:48.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851002 '!=' null 2024-04-25T08:21:48.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:48.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:48.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:48.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:48.912 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:48.912 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:48.912 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:48.913 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:48.914 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:48.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:49.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:49.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:49.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:49.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:49.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:49.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:49.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:49.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:49.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:49.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:49.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:49.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:49.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:49.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:49.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:49.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:49.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:49.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:49.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:49.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:49.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:50.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850810 2024-04-25T08:21:50.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850810 '!=' null 2024-04-25T08:21:50.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:50.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:50.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:50.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:50.848 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:50.848 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:50.849 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:50.850 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:50.851 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:50.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:50.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:50.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:50.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:50.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:50.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:50.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:51.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:51.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:51.430 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:51.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:51.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:51.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:51.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:51.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:51.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:51.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:51.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:51.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:51.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:52.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838931 2024-04-25T08:21:52.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838931 '!=' null 2024-04-25T08:21:52.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:52.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:52.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:52.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:52.643 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:52.643 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:52.643 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:52.644 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:52.645 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:52.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:52.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:52.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:52.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:52.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:52.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:52.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:52.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:52.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:53.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:53.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:53.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:53.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:53.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:53.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:53.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:53.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:53.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:53.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:53.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:53.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:53.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:54.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850071 2024-04-25T08:21:54.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850071 '!=' null 2024-04-25T08:21:54.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:54.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:54.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:54.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:54.535 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:54.536 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:54.537 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:54.538 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:54.538 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:54.538 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:54.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:54.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:54.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:54.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:54.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:54.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:54.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:54.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:54.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:55.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:55.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:55.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:55.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:55.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:55.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:55.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:55.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:55.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:55.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:55.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:55.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:55.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:56.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585458 2024-04-25T08:21:56.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585458 '!=' null 2024-04-25T08:21:56.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:56.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:56.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:56.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:56.416 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:56.416 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:56.416 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:56.416 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:56.417 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:56.418 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:56.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:56.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:56.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:56.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:56.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:56.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:56.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:56.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:56.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:56.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:56.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:56.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:56.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:57.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:57.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:21:57.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:58.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723786 2024-04-25T08:21:58.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723786 '!=' null 2024-04-25T08:21:58.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:21:58.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:58.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:21:58.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:21:58.376 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:58.377 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:21:58.378 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:21:58.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:21:58.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:21:58.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:21:58.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:21:58.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:21:58.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:21:58.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:21:58.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:21:58.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:21:58.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:21:58.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:21:58.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:21:58.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:21:59.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:21:59.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:21:59.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:21:59.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:21:59.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:21:59.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:21:59.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:21:59.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:21:59.573 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:00.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724103 2024-04-25T08:22:00.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724103 '!=' null 2024-04-25T08:22:00.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:00.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:00.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:00.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:00.321 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:00.322 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:00.323 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:00.323 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:00.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:00.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:00.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:00.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:00.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:00.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:00.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:00.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:00.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:00.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:00.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:00.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:00.834 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:01.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:01.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:01.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:02.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586961 2024-04-25T08:22:02.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586961 '!=' null 2024-04-25T08:22:02.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:02.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:02.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:02.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:02.121 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:02.121 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:02.122 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:02.123 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:02.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:02.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:02.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:02.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:02.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:02.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:02.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:02.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:02.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:02.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:02.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:02.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:02.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:03.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:03.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:03.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:03.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:03.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:03.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:03.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:03.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:03.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:03.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598277 2024-04-25T08:22:03.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598277 '!=' null 2024-04-25T08:22:03.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:03.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:03.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:03.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:03.904 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:03.904 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:03.904 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:03.904 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:03.904 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:03.904 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:03.904 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:03.904 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:03.904 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:03.904 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:03.904 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:03.905 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:03.906 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:03.906 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:03.906 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:03.906 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:03.906 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:03.906 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:03.906 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:03.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:04.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:04.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:04.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:04.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:04.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:04.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:04.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:04.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:04.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:04.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:04.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:04.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:05.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:05.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:05.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:05.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:05.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:05.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:05.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:05.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:05.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:05.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24461087 2024-04-25T08:22:05.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24461087 '!=' null 2024-04-25T08:22:05.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:05.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:05.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:05.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:05.981 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:05.981 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:05.981 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:05.981 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:05.982 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:05.982 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:05.982 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:05.982 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:05.982 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:05.982 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:05.982 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:05.982 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:05.982 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:05.982 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:05.982 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:05.983 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:05.984 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:05.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:06.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:06.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:06.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:06.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:06.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:06.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:06.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:06.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:06.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:06.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:06.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:06.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:07.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:07.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:07.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:07.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:07.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:07.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:07.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:07.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:07.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725128 2024-04-25T08:22:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725128 '!=' null 2024-04-25T08:22:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:07.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:07.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:07.945 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:07.945 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:07.945 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:07.945 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:07.946 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:07.947 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:07.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:08.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:08.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:08.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:08.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:08.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:08.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:08.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:08.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:08.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:08.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:08.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:08.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:09.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:09.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:09.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:09.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:09.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:09.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:09.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:09.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:09.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:09.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724972 2024-04-25T08:22:09.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724972 '!=' null 2024-04-25T08:22:09.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:09.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:09.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:09.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:09.836 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:09.836 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:09.836 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:09.837 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:09.838 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:09.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:09.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:09.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:09.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:09.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:09.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:09.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:09.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:09.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:10.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:10.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:10.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:10.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:10.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:10.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:10.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:10.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:10.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:10.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:10.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:10.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:10.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713457 2024-04-25T08:22:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713457 '!=' null 2024-04-25T08:22:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:11.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:11.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:11.572 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:11.677 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:11.677 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:11.677 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:11.677 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:11.677 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:11.678 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:11.679 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:11.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:11.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:11.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:11.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:11.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:11.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:11.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:11.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:11.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:12.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:12.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:12.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:12.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:12.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:12.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:12.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:12.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:12.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:12.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:12.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:12.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:12.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:13.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850940 2024-04-25T08:22:13.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850940 '!=' null 2024-04-25T08:22:13.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:13.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:13.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:13.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:13.559 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:13.560 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:13.561 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:13.562 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:13.562 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:13.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:13.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:13.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:13.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:13.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:13.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:13.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:13.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:13.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:14.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:14.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:14.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:14.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:14.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:14.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:14.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:14.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:14.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:14.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:14.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:14.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:14.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:15.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713614 2024-04-25T08:22:15.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713614 '!=' null 2024-04-25T08:22:15.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:15.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:15.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:15.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:15.347 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:15.347 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:15.347 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:15.347 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:15.347 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:15.347 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:15.347 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:15.347 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:15.347 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:15.347 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:15.348 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:15.349 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:15.349 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:15.349 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:15.349 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:15.349 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:15.349 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:15.349 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:15.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:15.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:15.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:15.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:15.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:15.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:15.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:15.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:15.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:15.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:15.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:15.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:15.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:16.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:16.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:16.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:16.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:16.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:16.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:16.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:16.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:16.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:16.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713614 2024-04-25T08:22:16.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713614 '!=' null 2024-04-25T08:22:16.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:16.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:16.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:16.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:17.062 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:17.062 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:17.062 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:17.062 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:17.062 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:17.062 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:17.062 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:17.062 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:17.062 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:17.063 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:17.064 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:17.064 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:17.064 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:17.064 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:17.064 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:17.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:17.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:17.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:17.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:17.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:17.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:17.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:17.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:17.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:17.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:17.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:17.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:17.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:18.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:18.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:18.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:18.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:18.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:18.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:18.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:18.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:18.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:18.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851004 2024-04-25T08:22:18.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851004 '!=' null 2024-04-25T08:22:18.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:18.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:18.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:18.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:18.793 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:18.793 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:18.793 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:18.793 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:18.793 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:18.793 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:18.793 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:18.793 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:18.793 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:18.793 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:18.794 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:18.795 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:18.795 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:18.795 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:18.795 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:18.795 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:18.795 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:18.795 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:18.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:18.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:18.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:18.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:18.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:18.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:18.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:18.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:18.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:19.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:19.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:19.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:19.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:19.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:19.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:19.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:19.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:19.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:19.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:19.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:19.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:19.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:20.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851190 2024-04-25T08:22:20.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851190 '!=' null 2024-04-25T08:22:20.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:20.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:20.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:20.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:20.796 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:20.796 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:20.796 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:20.796 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:20.796 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:20.797 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:20.798 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:20.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:20.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:20.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:20.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:20.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:20.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:20.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:20.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:20.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:21.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:21.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:21.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:21.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:21.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:21.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:21.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:21.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:21.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:21.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:21.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:21.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:21.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:22.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713785 2024-04-25T08:22:22.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713785 '!=' null 2024-04-25T08:22:22.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:22.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:22.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:22.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:22.681 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:22.682 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:22.683 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:22.683 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:22.683 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:22.683 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:22.683 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:22.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:22.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:22.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:22.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:22.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:22.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:22.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:22.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:22.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:23.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:23.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:23.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:23.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:23.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:23.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:23.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:23.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:23.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:23.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:23.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:23.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:23.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:24.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725118 2024-04-25T08:22:24.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725118 '!=' null 2024-04-25T08:22:24.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:24.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:24.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:24.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:24.366 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:24.366 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:24.366 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:24.367 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:24.368 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:24.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:24.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:24.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:24.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:24.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:24.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:24.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:24.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:24.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:24.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:24.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:24.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:24.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:25.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:25.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:25.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:25.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:25.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:25.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:25.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:25.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:25.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:26.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23587969 2024-04-25T08:22:26.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23587969 '!=' null 2024-04-25T08:22:26.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:26.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:26.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:26.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:26.303 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:26.304 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:26.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:26.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:26.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:26.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:26.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:26.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:26.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:26.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:26.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:26.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:26.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:26.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:26.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:27.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:27.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:27.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:27.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:27.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:27.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:27.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:27.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:27.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:28.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725046 2024-04-25T08:22:28.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725046 '!=' null 2024-04-25T08:22:28.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:28.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:28.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:28.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:28.131 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:28.131 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:28.131 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:28.131 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:28.131 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:28.131 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:28.131 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:28.132 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:28.133 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:28.133 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:28.133 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:28.133 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:28.133 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:28.133 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:28.133 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:28.133 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:28.133 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:28.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:28.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:28.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:28.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:28.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:28.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:28.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:28.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:28.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:28.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:28.640 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:28.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:28.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:29.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:29.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:29.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:29.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:29.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:29.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:29.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:29.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:29.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:29.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725298 2024-04-25T08:22:29.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725298 '!=' null 2024-04-25T08:22:29.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:29.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:29.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:29.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:29.900 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:29.900 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:29.900 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:29.900 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:29.900 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:29.900 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:29.900 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:29.900 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:29.900 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:29.900 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:29.901 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:29.902 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:29.902 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:29.902 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:29.902 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:29.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:30.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:30.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:30.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:30.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:30.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:30.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:30.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:30.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:30.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:30.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:30.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:30.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:31.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:31.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:31.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:31.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:31.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:31.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:31.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:31.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:31.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:31.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23587124 2024-04-25T08:22:31.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23587124 '!=' null 2024-04-25T08:22:31.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:31.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:31.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:31.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:31.934 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:31.934 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:31.934 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:31.935 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:31.936 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:31.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:32.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:32.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:32.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:32.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:32.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:32.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:32.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:32.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:32.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:32.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:32.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:32.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:33.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:33.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:33.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:33.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:33.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:33.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:33.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:33.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:33.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:33.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724579 2024-04-25T08:22:33.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724579 '!=' null 2024-04-25T08:22:33.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:33.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:33.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:33.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:33.833 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:33.833 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:33.834 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:33.835 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:33.836 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:33.836 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:33.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:33.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:33.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:33.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:33.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:33.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:33.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:33.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:33.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:34.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:34.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:34.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:34.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:34.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:34.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:34.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:34.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:34.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:34.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:34.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:34.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:34.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586976 2024-04-25T08:22:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586976 '!=' null 2024-04-25T08:22:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:35.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:35.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:35.566 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:35.566 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:35.567 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:35.568 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:35.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:35.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:35.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:35.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:35.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:35.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:35.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:35.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:35.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:36.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:36.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:36.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:36.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:36.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:36.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:36.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:36.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:36.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:36.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:36.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:36.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:36.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:37.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586976 2024-04-25T08:22:37.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586976 '!=' null 2024-04-25T08:22:37.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:37.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:37.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:37.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:37.336 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:37.336 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:37.337 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:37.338 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:37.339 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:37.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:37.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:37.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:37.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:37.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:37.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:37.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:37.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:37.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:38.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:38.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:38.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:38.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:38.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:38.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:38.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:38.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:38.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:38.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:38.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:38.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:38.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:39.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724186 2024-04-25T08:22:39.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724186 '!=' null 2024-04-25T08:22:39.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:39.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:39.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:39.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:39.532 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:39.532 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:39.532 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:39.533 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:39.534 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:39.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:39.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:39.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:39.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:39.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:39.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:39.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:39.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:40.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:40.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:40.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:40.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:40.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:40.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:40.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:40.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:40.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:40.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:40.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:40.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:40.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:41.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712620 2024-04-25T08:22:41.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712620 '!=' null 2024-04-25T08:22:41.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:41.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:41.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:41.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:41.384 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:41.385 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:41.386 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:41.386 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:41.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:41.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:41.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:41.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:41.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:41.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:41.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:41.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:41.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:41.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:41.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:41.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:41.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:42.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:42.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:42.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:42.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:42.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:42.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:42.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:42.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:42.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:43.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712620 2024-04-25T08:22:43.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712620 '!=' null 2024-04-25T08:22:43.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:43.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:43.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:43.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:43.268 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:43.268 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:43.268 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:43.268 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:43.268 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:43.268 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:43.268 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:43.268 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:43.269 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:43.270 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:43.270 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:43.270 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:43.270 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:43.270 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:43.270 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:43.270 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:43.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:43.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:43.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:43.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:43.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:43.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:43.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:43.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:43.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:43.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:43.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:43.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:43.832 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:44.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:44.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:44.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:44.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:44.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:44.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:44.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:44.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:44.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:44.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850745 2024-04-25T08:22:44.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850745 '!=' null 2024-04-25T08:22:44.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:44.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:44.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:44.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:45.005 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:45.005 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:45.005 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:45.005 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:45.005 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:45.005 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:45.005 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:45.005 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:45.005 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:45.005 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:45.006 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:45.007 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:45.007 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:45.007 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:45.007 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:45.007 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:45.007 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:45.007 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:45.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:45.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:45.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:45.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:45.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:45.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:45.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:45.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:45.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:45.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:45.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:45.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:45.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:46.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:46.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:46.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:46.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:46.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:46.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:46.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:46.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:46.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:46.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586618 2024-04-25T08:22:46.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586618 '!=' null 2024-04-25T08:22:46.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:46.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:46.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:46.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:46.897 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:46.898 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:46.899 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:46.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:47.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:47.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:47.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:47.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:47.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:47.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:47.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:47.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:47.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:47.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:47.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:47.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:48.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:48.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:48.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:48.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:48.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:48.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:48.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:48.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:48.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:48.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724311 2024-04-25T08:22:48.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724311 '!=' null 2024-04-25T08:22:48.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:48.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:48.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:48.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:48.783 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:48.783 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:48.783 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:48.783 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:48.784 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:48.785 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:48.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:48.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:48.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:48.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:48.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:48.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:48.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:48.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:48.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:49.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:49.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:49.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:49.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:50.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:50.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:50.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:50.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:50.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:50.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:50.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:50.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:50.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:50.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724213 2024-04-25T08:22:50.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724213 '!=' null 2024-04-25T08:22:50.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:50.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:50.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:50.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:50.962 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:50.962 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [ 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 190, 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T08:22:50.963 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "1.0", 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:50.964 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:50.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:51.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:51.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:51.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:51.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:51.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:51.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:51.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:51.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:51.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:51.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:51.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:51.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:52.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:52.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:52.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:52.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:52.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:52.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:52.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:52.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:52.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:53.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10482494 2024-04-25T08:22:53.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10482494 '!=' null 2024-04-25T08:22:53.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:53.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:53.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:53.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:53.766 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:53.767 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:53.767 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:53.767 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:53.767 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:53.767 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:53.767 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:53.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:53.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:53.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:53.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:53.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:53.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:53.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:53.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:53.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:54.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:22:54.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:54.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:54.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:54.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:22:54.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:22:54.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:54.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:54.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:54.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:54.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:54.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:54.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:55.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23584747 2024-04-25T08:22:55.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23584747 '!=' null 2024-04-25T08:22:55.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:55.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:55.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:55.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:55.978 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:55.979 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:55.979 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:55.979 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:55.979 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:55.979 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:55.979 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:55.979 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:55.979 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:55.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:56.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:56.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:56.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:56.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:56.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:56.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:56.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:56.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:56.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:22:56.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:56.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:56.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:57.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:22:57.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 1 2024-04-25T08:22:57.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:22:57.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-04-25T08:22:57.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:57.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:57.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:57.134 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:57.134 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:57.135 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:57.135 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:57.135 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:57.135 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:57.135 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:57.135 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:57.135 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:57.135 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:57.135 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:57.135 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:57.135 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:57.136 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:57.136 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:57.136 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:57.136 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:57.136 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:57.136 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:57.136 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:57.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:57.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:57.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:57.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:57.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:57.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:57.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:57.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:57.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:57.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:22:57.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:57.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:22:57.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:58.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:22:58.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:22:58.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:22:58.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:22:58.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:22:58.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:22:58.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:22:58.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:22:58.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:22:58.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23585482 2024-04-25T08:22:58.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23585482 '!=' null 2024-04-25T08:22:58.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:22:58.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:58.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:22:58.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:22:59.039 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:22:59.039 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:22:59.039 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:59.039 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:59.039 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:59.039 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:22:59.039 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:22:59.039 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:22:59.039 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:22:59.039 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:22:59.040 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:59.040 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:22:59.040 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:22:59.040 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:22:59.040 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:22:59.040 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:22:59.040 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:22:59.040 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:22:59.040 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:22:59.040 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:22:59.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:22:59.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:22:59.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:22:59.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:22:59.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:22:59.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:22:59.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:22:59.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:22:59.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:22:59.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:22:59.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:22:59.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:22:59.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:00.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:00.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:00.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:00.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:00.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:00.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:00.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:00.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:00.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:00.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13102889 2024-04-25T08:23:00.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13102889 '!=' null 2024-04-25T08:23:00.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:00.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:00.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:00.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:00.797 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:00.797 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:00.798 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:00.799 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:00.799 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:00.799 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:00.799 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:00.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:00.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:00.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:00.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:00.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:00.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:00.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:00.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:00.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:01.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:01.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:01.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:01.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:02.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:02.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:02.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:02.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:02.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:02.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:02.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:02.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:02.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23995128 2024-04-25T08:23:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23995128 '!=' null 2024-04-25T08:23:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:02.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:02.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:02.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:02.900 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:02.900 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:02.900 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:02.900 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:02.900 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:02.900 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:02.901 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:02.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:03.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:03.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:03.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:03.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:03.014 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:03.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:03.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:03.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:03.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:03.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:03.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:03.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:04.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:04.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:04.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:04.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:04.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:04.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:04.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:04.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:04.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:04.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21150106 2024-04-25T08:23:04.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21150106 '!=' null 2024-04-25T08:23:04.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:04.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:04.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:04.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:04.901 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:04.901 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:04.901 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:04.901 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:04.901 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:04.901 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:04.901 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:04.902 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:04.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:05.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:05.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:05.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:05.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:05.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:05.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:05.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:05.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:05.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:05.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:05.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:05.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:06.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:06.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:06.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26206709 2024-04-25T08:23:06.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26206709 '!=' null 2024-04-25T08:23:06.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:06.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:06.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:06.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:06.819 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:06.820 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:06.821 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:06.821 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:06.821 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:06.821 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:06.821 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:06.821 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:06.821 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:06.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:06.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:06.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:06.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:06.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:06.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:06.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:06.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:06.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:07.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:07.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:07.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:07.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:07.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:07.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:07.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:07.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:07.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:07.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:07.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:07.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:07.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:08.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471486 2024-04-25T08:23:08.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471486 '!=' null 2024-04-25T08:23:08.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:08.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:08.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:08.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:08.702 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:08.702 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:08.702 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:08.702 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:08.703 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:08.703 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:08.703 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:08.703 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:08.703 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:08.703 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:08.704 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:08.704 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:08.704 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:08.704 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:08.704 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:08.704 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:08.705 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:08.705 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:08.705 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:08.705 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:08.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:08.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:08.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:08.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:08.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:08.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:08.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:08.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:08.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:09.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:09.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:09.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:09.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:09.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:09.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:09.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:09.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:09.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:09.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:09.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:09.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:10.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471474 2024-04-25T08:23:10.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471474 '!=' null 2024-04-25T08:23:10.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:10.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:10.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:10.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:10.474 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:10.474 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:10.474 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:10.474 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:10.475 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:10.476 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:10.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:10.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:10.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:10.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:10.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:10.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:10.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:10.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:10.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:11.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:11.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:11.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:11.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:11.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:11.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:11.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:11.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:11.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:11.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:11.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:11.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:11.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:12.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24460874 2024-04-25T08:23:12.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24460874 '!=' null 2024-04-25T08:23:12.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:12.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:12.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:12.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:12.438 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:12.454 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:12.454 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:12.454 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:12.454 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:12.454 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:12.454 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:12.455 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:12.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:12.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:12.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:12.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:12.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:12.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:12.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:12.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:12.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:12.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:12.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:12.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:12.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:13.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:13.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:13.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:13.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:13.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:13.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:13.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:13.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:13.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:14.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851268 2024-04-25T08:23:14.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851268 '!=' null 2024-04-25T08:23:14.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:14.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:14.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:14.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:14.167 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:14.167 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:14.167 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:14.167 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:14.167 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:14.167 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:14.167 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:14.167 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:14.167 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:14.167 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:14.168 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:14.168 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:14.168 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:14.168 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:14.168 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:14.168 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:14.168 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:14.168 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:14.168 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:14.168 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:14.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:14.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:14.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:14.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:14.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:14.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:14.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:14.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:14.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:14.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:14.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:14.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:14.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:15.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:15.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:15.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:15.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:15.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:15.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:15.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:15.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:15.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:16.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22714002 2024-04-25T08:23:16.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22714002 '!=' null 2024-04-25T08:23:16.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:16.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:16.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:16.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:16.188 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:16.189 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:16.190 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:16.190 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:16.190 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:16.190 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:16.190 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:16.190 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:16.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:16.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:16.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:16.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:16.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:16.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:16.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:16.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:16.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:16.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:16.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:16.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:16.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:17.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:17.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:17.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:17.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:17.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:17.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:17.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:17.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:17.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:17.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851278 2024-04-25T08:23:17.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851278 '!=' null 2024-04-25T08:23:17.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:17.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:17.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:17.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:18.002 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:18.002 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:18.002 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:18.002 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:18.002 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:18.002 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:18.003 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:18.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:18.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:18.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:18.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:18.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:18.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:18.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:18.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:18.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:18.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:18.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:18.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:18.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:19.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:19.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:19.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:19.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:19.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:19.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:19.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:19.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:19.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:20.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851031 2024-04-25T08:23:20.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851031 '!=' null 2024-04-25T08:23:20.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:20.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:20.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:20.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:20.135 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:20.136 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:20.137 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:20.137 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:20.137 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:20.137 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:20.137 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:20.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:20.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:20.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:20.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:20.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:20.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:20.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:20.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:20.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:20.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:20.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:20.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:21.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:21.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:21.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:21.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:21.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:21.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:21.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:21.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:21.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:21.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586700 2024-04-25T08:23:21.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586700 '!=' null 2024-04-25T08:23:21.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:21.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:21.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:21.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:21.975 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:21.975 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:21.975 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:21.975 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:21.975 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:21.975 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:21.975 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:21.975 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:21.976 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:21.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:22.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:22.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:22.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:22.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:22.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:22.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:22.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:22.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:22.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:22.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:22.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:22.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:23.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:23.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:23.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:23.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:23.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:23.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:23.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:23.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:23.179 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:23.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597438 2024-04-25T08:23:23.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597438 '!=' null 2024-04-25T08:23:23.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:23.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:23.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:23.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:23.981 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:23.981 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:23.981 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:23.981 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:23.982 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:23.983 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:23.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:24.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:24.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:24.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:24.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:24.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:24.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:24.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:24.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:24.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:24.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:24.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:24.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:25.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:25.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:25.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:25.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:25.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:25.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:25.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:25.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:25.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:25.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459353 2024-04-25T08:23:25.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459353 '!=' null 2024-04-25T08:23:25.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:25.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:25.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:25.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:26.054 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:26.055 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:26.055 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:26.055 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:26.055 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:26.055 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:26.055 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:26.056 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:26.056 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:26.056 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:26.056 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:26.056 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:26.056 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:26.056 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:26.056 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:26.056 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:26.056 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:26.057 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:26.057 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:26.057 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:26.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:26.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:26.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:26.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:26.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:26.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:26.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:26.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:26.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:26.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:26.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:26.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:26.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:27.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:27.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:27.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:27.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:27.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:27.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:27.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:27.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:27.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:27.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597244 2024-04-25T08:23:27.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597244 '!=' null 2024-04-25T08:23:27.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:27.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:27.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:27.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:27.800 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:27.800 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:27.800 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:27.801 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:27.802 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:27.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:27.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:27.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:27.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:27.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:27.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:27.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:27.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:27.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:28.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:28.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:28.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:28.320 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:28.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:28.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:28.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:28.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:28.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:28.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:28.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:28.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:28.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:29.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597161 2024-04-25T08:23:29.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597161 '!=' null 2024-04-25T08:23:29.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:29.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:29.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:29.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:29.604 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:29.605 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:29.605 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:29.605 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:29.605 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:29.605 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:29.605 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:29.605 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:29.605 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:29.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:29.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:29.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:29.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:29.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:29.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:29.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:29.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:29.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:30.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:30.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:30.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:30.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:30.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:30.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:30.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:30.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:30.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:30.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:30.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:30.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:30.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:31.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459265 2024-04-25T08:23:31.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459265 '!=' null 2024-04-25T08:23:31.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:31.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:31.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:31.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:31.480 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:31.481 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:31.482 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:31.482 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:31.482 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:31.482 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:31.482 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:31.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:31.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:31.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:31.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:31.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:31.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:31.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:31.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:31.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:32.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:32.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:32.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:32.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:32.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:32.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:32.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:32.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:33.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459265 2024-04-25T08:23:33.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459265 '!=' null 2024-04-25T08:23:33.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:33.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:33.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:33.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:33.377 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:33.377 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:33.377 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:33.377 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:33.378 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:33.378 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:33.378 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:33.378 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:33.378 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:33.378 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:33.378 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:33.378 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:33.378 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:33.378 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:33.378 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:33.379 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:33.379 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:33.379 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:33.379 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:33.379 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:33.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:33.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:33.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:33.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:33.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:33.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:33.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:33.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:33.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:33.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:33.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:33.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:33.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:34.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:34.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:34.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:34.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:34.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:34.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:34.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:34.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:34.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:35.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723929 2024-04-25T08:23:35.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723929 '!=' null 2024-04-25T08:23:35.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:35.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:35.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:35.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:35.336 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:35.336 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:35.336 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:35.336 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:35.336 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:35.336 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:35.336 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:35.336 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:35.336 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:35.336 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:35.337 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:35.337 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:35.337 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:35.337 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:35.337 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:35.337 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:35.337 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:35.337 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:35.337 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:35.337 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:35.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:35.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:35.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:35.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:35.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:35.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:35.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:35.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:35.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:35.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:35.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:35.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:35.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:36.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:36.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:36.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:36.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:36.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:36.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:36.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:36.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:36.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:36.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712653 2024-04-25T08:23:36.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712653 '!=' null 2024-04-25T08:23:36.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:36.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:36.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:36.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:37.057 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:37.057 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:37.057 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:37.057 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:37.057 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:37.057 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:37.058 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:37.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:37.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:37.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:37.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:37.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:37.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:37.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:37.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:37.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:37.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:37.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:37.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:37.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:38.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:38.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:38.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:38.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:38.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850602 2024-04-25T08:23:38.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850602 '!=' null 2024-04-25T08:23:38.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:38.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:38.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:38.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:38.976 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:38.976 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:38.976 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:38.976 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:38.976 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:38.976 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:38.976 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:38.976 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:38.976 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:38.977 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:38.977 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:38.977 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:38.977 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:38.977 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:38.977 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:38.977 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:38.977 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:38.977 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:38.977 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:38.977 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:38.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:39.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:39.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:39.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:39.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:39.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:39.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:39.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:39.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:39.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:39.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:39.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:39.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:40.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:40.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:40.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:40.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:40.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:40.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:40.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:40.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:40.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:40.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850858 2024-04-25T08:23:40.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850858 '!=' null 2024-04-25T08:23:40.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:40.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:40.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:40.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:41.063 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:41.063 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:41.063 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:41.063 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:41.063 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:41.063 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:41.063 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:41.063 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:41.063 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:41.063 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:41.063 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:41.064 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:41.064 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:41.064 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:41.064 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:41.064 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:41.064 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:41.064 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:41.064 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:41.064 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:41.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:41.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:41.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:41.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:41.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:41.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:41.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:41.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:41.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:41.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:41.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:41.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:41.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:42.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:42.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:42.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:42.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:42.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:42.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:42.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:42.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:42.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:42.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713150 2024-04-25T08:23:42.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713150 '!=' null 2024-04-25T08:23:42.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:42.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:42.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:42.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:42.946 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:42.946 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:42.946 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:42.946 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:42.946 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:42.947 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:42.947 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:42.947 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:42.947 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:42.947 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:42.947 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:42.947 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:42.947 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:42.947 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:42.948 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:42.948 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:42.948 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:42.948 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:42.948 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:42.948 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:42.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:43.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:43.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:43.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:43.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:43.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:43.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:43.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:43.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:43.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:43.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:43.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:43.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:44.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:44.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:44.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:44.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:44.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:44.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:44.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:44.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:44.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:44.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851108 2024-04-25T08:23:44.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851108 '!=' null 2024-04-25T08:23:44.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:44.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:44.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:44.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:44.874 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:44.874 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:44.874 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:44.874 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:44.874 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:44.874 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:44.874 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:44.874 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:44.874 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:44.875 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:44.875 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:44.875 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:44.875 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:44.875 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:44.875 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:44.875 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:44.875 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:44.875 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:44.875 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:44.875 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:44.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:44.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:44.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:44.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:44.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:44.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:44.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:44.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:44.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:45.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:45.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:45.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:45.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:45.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:45.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:45.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:45.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:45.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:45.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:45.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:45.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:45.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:46.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839682 2024-04-25T08:23:46.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839682 '!=' null 2024-04-25T08:23:46.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:46.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:46.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:46.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:46.703 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:46.703 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:46.703 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:46.703 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:46.703 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:46.703 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:46.703 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:46.703 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:46.704 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:46.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:46.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:46.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:46.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:46.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:46.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:46.817 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:46.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:46.818 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:47.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:47.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:47.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:47.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:47.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:47.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:47.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:47.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:47.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:47.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:47.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:47.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:47.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:48.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850974 2024-04-25T08:23:48.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850974 '!=' null 2024-04-25T08:23:48.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:48.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:48.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:48.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:48.538 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:48.538 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:48.538 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:48.538 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:48.538 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:48.538 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:48.538 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:48.538 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:48.538 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:48.538 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:48.539 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:48.539 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:48.539 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:48.539 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:48.539 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:48.539 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:48.539 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:48.539 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:48.539 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:48.539 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:48.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:48.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:48.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:48.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:48.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:48.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:48.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:48.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:48.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:49.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:49.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:49.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:49.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:49.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:49.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:49.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:49.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:49.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:49.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:49.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:49.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:49.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851220 2024-04-25T08:23:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851220 '!=' null 2024-04-25T08:23:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:50.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:50.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:50.429 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:50.538 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:50.539 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:50.540 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:50.540 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:50.540 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:50.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:50.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:50.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:50.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:50.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:50.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:50.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:50.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:50.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:51.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:51.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:51.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:51.076 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:51.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:51.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:51.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:51.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:51.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:51.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:51.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:51.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:51.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:52.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713256 2024-04-25T08:23:52.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713256 '!=' null 2024-04-25T08:23:52.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:52.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:52.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:52.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:52.352 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:52.353 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:52.354 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:52.354 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:52.354 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:52.354 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:52.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:52.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:52.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:52.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:52.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:52.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:52.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:52.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:52.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:52.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:52.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:52.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:52.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:53.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:53.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:53.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:53.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:53.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:53.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:53.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:53.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:53.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:54.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850931 2024-04-25T08:23:54.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850931 '!=' null 2024-04-25T08:23:54.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:54.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:54.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:54.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:54.222 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:54.222 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:54.222 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:54.223 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:54.224 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:54.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:54.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:54.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:54.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:54.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:54.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:54.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:54.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:54.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:54.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:54.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:54.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:54.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:55.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:55.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:55.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:55.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:55.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:55.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:55.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:55.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:55.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:55.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713237 2024-04-25T08:23:55.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713237 '!=' null 2024-04-25T08:23:55.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:55.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:55.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:55.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:56.105 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:56.105 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:56.105 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:56.105 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:56.105 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:56.105 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:56.105 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:56.105 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:56.105 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:56.106 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:56.106 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:56.106 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:56.106 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:56.106 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:56.106 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:56.106 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:56.106 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:56.106 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:56.106 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:56.106 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:56.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:56.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:56.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:56.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:56.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:56.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:56.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:56.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:56.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:56.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:56.643 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:56.643 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:56.643 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:57.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:57.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:57.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:57.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:57.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:57.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:57.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:57.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:57.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:57.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724548 2024-04-25T08:23:57.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724548 '!=' null 2024-04-25T08:23:57.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:57.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:57.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:57.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:57.912 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:57.912 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:57.913 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:57.914 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:57.914 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:57.914 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:57.914 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:57.914 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:57.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:58.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:58.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:58.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:58.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:58.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:58.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:58.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:58.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:23:58.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:23:58.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:23:58.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:23:58.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:23:59.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:23:59.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:23:59.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:23:59.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:23:59.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:23:59.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:23:59.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:23:59.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:23:59.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:23:59.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724576 2024-04-25T08:23:59.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724576 '!=' null 2024-04-25T08:23:59.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:23:59.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:59.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:23:59.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:23:59.748 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:23:59.748 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:23:59.748 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:59.749 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:59.749 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:59.749 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:23:59.749 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:23:59.749 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:23:59.749 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:23:59.749 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:23:59.750 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:59.750 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:23:59.750 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:23:59.750 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:23:59.750 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:23:59.750 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:23:59.751 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:23:59.751 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:23:59.751 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:23:59.751 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:23:59.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:23:59.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:23:59.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:23:59.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:23:59.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:23:59.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:23:59.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:23:59.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:23:59.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:00.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:00.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:00.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:00.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:01.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:01.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:01.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:01.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:01.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:01.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:01.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:01.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:01.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:01.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586386 2024-04-25T08:24:01.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586386 '!=' null 2024-04-25T08:24:01.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:01.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:01.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:01.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:01.954 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:01.954 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:01.954 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:01.955 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:01.956 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:01.956 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:01.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:02.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:02.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:02.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:02.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:02.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:02.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:02.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:02.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:02.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:02.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:02.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:02.474 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:03.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:03.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:03.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:03.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:03.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:03.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:03.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:03.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:03.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:03.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724216 2024-04-25T08:24:03.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724216 '!=' null 2024-04-25T08:24:03.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:03.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:03.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:03.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:03.788 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:03.789 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:03.789 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:03.789 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:03.789 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:03.789 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:03.789 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:03.789 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:03.789 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:03.789 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:03.789 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:03.789 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:03.790 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:03.790 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:03.790 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:03.790 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:03.790 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:03.790 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:03.790 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:03.790 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:03.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:03.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:03.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:03.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:03.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:03.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:03.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:03.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:03.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:04.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:04.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:04.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:04.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:04.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:04.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:04.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:04.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:04.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:04.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:04.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:04.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:04.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:05.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586315 2024-04-25T08:24:05.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586315 '!=' null 2024-04-25T08:24:05.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:05.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:05.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:05.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:05.859 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:05.859 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:05.859 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:05.859 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:05.859 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:05.859 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:05.859 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:05.860 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:05.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:05.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:05.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:05.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:05.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:05.974 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:05.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:05.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:05.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:06.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:06.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:06.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:06.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:07.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:07.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:07.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:07.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:07.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:07.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:07.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:07.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:07.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724038 2024-04-25T08:24:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724038 '!=' null 2024-04-25T08:24:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:07.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:07.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:07.752 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:07.752 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:07.753 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:07.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:07.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:07.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:07.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:07.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:07.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:07.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:07.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:07.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:08.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:08.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:08.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:08.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:08.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:08.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:08.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:08.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:08.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:08.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:08.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:08.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:08.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724152 2024-04-25T08:24:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724152 '!=' null 2024-04-25T08:24:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:09.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:09.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:09.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:09.615 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:09.615 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:09.616 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:09.617 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:09.617 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:09.617 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:09.617 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:09.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:09.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:09.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:09.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:09.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:09.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:09.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:09.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:09.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:10.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:10.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:10.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:10.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:11.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:11.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:11.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:11.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586538 2024-04-25T08:24:11.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586538 '!=' null 2024-04-25T08:24:11.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:11.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:11.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:11.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:12.004 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:12.004 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:12.004 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:12.004 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:12.004 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:12.004 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:12.004 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:12.004 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:12.004 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:12.004 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:12.004 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:12.005 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:12.005 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:12.005 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:12.005 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:12.005 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:12.005 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:12.005 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:12.005 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:12.005 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:12.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:12.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:12.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:12.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:12.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:12.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:12.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:12.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:12.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:12.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:12.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:12.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:12.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:13.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:13.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:13.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:13.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:13.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:13.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:13.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:13.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:13.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:13.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724371 2024-04-25T08:24:13.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724371 '!=' null 2024-04-25T08:24:13.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:13.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:13.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:13.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:13.964 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:13.964 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:13.964 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:13.964 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:13.964 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:13.964 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:13.964 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:13.964 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:13.964 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:13.965 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:13.965 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:13.965 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:13.965 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:13.965 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:13.965 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:13.965 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:13.965 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:13.965 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:13.965 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:13.965 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:13.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:14.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:14.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:14.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:14.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:14.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:14.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:14.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:14.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:14.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:14.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:14.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:14.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:15.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:15.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:15.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:15.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:15.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:15.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:15.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:15.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:15.242 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:15.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586766 2024-04-25T08:24:15.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586766 '!=' null 2024-04-25T08:24:15.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:15.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:15.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:15.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:15.887 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:15.887 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:15.887 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:15.888 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:15.889 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:15.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:16.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:16.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:16.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:16.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:16.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:16.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:16.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:16.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:16.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:16.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:16.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:16.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:17.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:17.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:17.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:17.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:17.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:17.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:17.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:17.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:17.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:17.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724620 2024-04-25T08:24:17.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724620 '!=' null 2024-04-25T08:24:17.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:17.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:17.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:17.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:17.856 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:17.856 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:17.856 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:17.857 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:17.858 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:17.858 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:17.858 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:17.858 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:17.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:17.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:17.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:17.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:17.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:17.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:17.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:17.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:17.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:18.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:18.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:18.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:18.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:18.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:18.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:18.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:18.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:18.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:18.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:18.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:18.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:18.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:19.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724575 2024-04-25T08:24:19.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724575 '!=' null 2024-04-25T08:24:19.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:19.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:19.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:19.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:19.599 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:19.600 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:19.600 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:19.600 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:19.600 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:19.600 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:19.600 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:19.600 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:19.600 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:19.600 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:19.600 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:19.600 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:19.601 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:19.601 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:19.601 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:19.601 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:19.601 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:19.601 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:19.601 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:19.601 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:19.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:19.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:19.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:19.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:19.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:19.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:19.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:19.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:19.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:20.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:20.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:20.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:20.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:20.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:20.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:20.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:20.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:20.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:20.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:20.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:20.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:20.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:21.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586836 2024-04-25T08:24:21.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586836 '!=' null 2024-04-25T08:24:21.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:21.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:21.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:21.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:21.356 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:21.357 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:21.358 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:21.358 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:21.358 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:21.358 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:21.358 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:21.358 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:21.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:21.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:21.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:21.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:21.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:21.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:21.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:21.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:21.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:21.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:21.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:21.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:21.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:22.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:22.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:22.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:22.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:22.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:22.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:22.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:22.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:22.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:23.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586836 2024-04-25T08:24:23.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586836 '!=' null 2024-04-25T08:24:23.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:23.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:23.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:23.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:23.387 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:23.387 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:23.387 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:23.387 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:23.387 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:23.388 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:23.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:23.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:23.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:23.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:23.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:23.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:23.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:23.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:23.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:23.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:23.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:23.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:24.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:24.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:24.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:24.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:24.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:24.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:24.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:24.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:24.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:25.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724474 2024-04-25T08:24:25.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724474 '!=' null 2024-04-25T08:24:25.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:25.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:25.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:25.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:25.174 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:25.174 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:25.174 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:25.174 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:25.174 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:25.174 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:25.174 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:25.174 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:25.174 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:25.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:25.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:25.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:25.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:25.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:25.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:25.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:25.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:25.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:25.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:25.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:25.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:25.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:26.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:26.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:26.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:26.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:26.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:26.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:26.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:26.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:26.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:26.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586899 2024-04-25T08:24:26.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586899 '!=' null 2024-04-25T08:24:26.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:26.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:26.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:26.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:26.935 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:26.936 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:26.937 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:26.937 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:26.937 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:26.937 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:26.937 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:26.937 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:26.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:27.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:27.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:27.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:27.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:27.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:27.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:27.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:27.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:27.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:27.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:27.491 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:28.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:28.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:28.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:28.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:28.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:28.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:28.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:28.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:28.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:28.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724586 2024-04-25T08:24:28.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724586 '!=' null 2024-04-25T08:24:28.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:28.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:28.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:28.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:28.915 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:28.915 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:28.915 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:28.915 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:28.916 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:28.917 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:28.917 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:28.917 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:28.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:29.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:29.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:29.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:29.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:29.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:29.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:29.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:29.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:29.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:29.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:29.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:29.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:30.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:30.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:30.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:30.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:30.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:30.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:30.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:30.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:30.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:30.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724432 2024-04-25T08:24:30.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724432 '!=' null 2024-04-25T08:24:30.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:30.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:30.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:30.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:30.841 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:30.841 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:30.841 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:30.841 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:30.841 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:30.841 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:30.841 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:30.841 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:30.841 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:30.842 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:30.842 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:30.842 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:30.842 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:30.842 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:30.842 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:30.842 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:30.842 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:30.842 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:30.842 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:30.842 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:30.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:30.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:30.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:30.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:30.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:30.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:30.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:30.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:30.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:31.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:31.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:31.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:31.457 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:32.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:32.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:32.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:32.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:32.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:32.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:32.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:32.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:32.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:32.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586490 2024-04-25T08:24:32.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586490 '!=' null 2024-04-25T08:24:32.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:32.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:32.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:32.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:32.646 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:32.647 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:32.648 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:32.648 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:32.648 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:32.648 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:32.648 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:32.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:32.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:32.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:32.760 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:32.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:32.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:32.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:32.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:32.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:33.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:33.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:33.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:33.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:33.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:33.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:33.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:33.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:33.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:33.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:33.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:33.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:33.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:34.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724148 2024-04-25T08:24:34.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724148 '!=' null 2024-04-25T08:24:34.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:34.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:34.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:34.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:34.543 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:34.544 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:34.544 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:34.544 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:34.544 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:34.544 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:34.544 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:34.544 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:34.544 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:34.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:34.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:34.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:34.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:34.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:34.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:34.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:34.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:34.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:35.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:35.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:35.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:35.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:35.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:35.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:35.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:35.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:35.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:35.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:35.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:35.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:35.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:36.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586064 2024-04-25T08:24:36.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586064 '!=' null 2024-04-25T08:24:36.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:36.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:36.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:36.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:36.522 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:36.522 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:36.522 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:36.522 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:36.522 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:36.523 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:36.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:36.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:36.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:36.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:36.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:36.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:36.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:36.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:36.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:37.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:37.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:37.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:37.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:37.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:37.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:37.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:37.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:37.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:37.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:37.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:37.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:37.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:38.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724225 2024-04-25T08:24:38.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724225 '!=' null 2024-04-25T08:24:38.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:38.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:38.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:38.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:38.409 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:38.409 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:38.410 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:38.411 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:38.411 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:38.411 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:38.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:38.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:38.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:38.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:38.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:38.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:38.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:38.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:38.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:38.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:38.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:38.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:38.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:39.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:39.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:39.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:39.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:39.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:39.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:39.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:39.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:39.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:40.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724086 2024-04-25T08:24:40.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724086 '!=' null 2024-04-25T08:24:40.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:40.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:40.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:40.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:40.437 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:40.437 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:40.437 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:40.437 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:40.437 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:40.437 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [ 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: "item": "3.0", 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: "prio": 190, 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: "can_preempt": true 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:40.438 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:40.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:40.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:40.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:40.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:40.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:40.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:40.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:40.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:40.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:40.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:40.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:40.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:40.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:41.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:41.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:41.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:41.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:41.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:41.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:41.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:41.643 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:41.643 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:42.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586215 2024-04-25T08:24:42.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586215 '!=' null 2024-04-25T08:24:42.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:42.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:42.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:42.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:42.436 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:42.436 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:42.436 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:42.436 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:42.436 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:42.436 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:42.437 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:42.437 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:42.437 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:42.437 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:42.437 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:42.437 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:42.437 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:42.437 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:42.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:42.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:42.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:42.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:42.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:42.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:42.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:42.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:42.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:43.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:43.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:43.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:43.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:43.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:43.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:43.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:43.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:43.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:43.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:43.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:43.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:43.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:44.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18869531 2024-04-25T08:24:44.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18869531 '!=' null 2024-04-25T08:24:44.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:44.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:44.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:44.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:44.276 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:44.277 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:44.277 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:44.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:44.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:44.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:44.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:44.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:44.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:44.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:44.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:44.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:44.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:24:44.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:44.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:44.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:45.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:24:45.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:24:45.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:24:45.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:24:45.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:24:45.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:24:45.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:24:45.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:24:45.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:24:45.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18345095 2024-04-25T08:24:45.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18345095 '!=' null 2024-04-25T08:24:45.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:24:45.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:45.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:24:45.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:24:45.983 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:45.983 INFO:tasks.workunit.client.0.smithi102.stdout: "local_reservations": { 2024-04-25T08:24:45.983 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:45.983 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:45.983 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:45.983 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:45.983 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:45.983 INFO:tasks.workunit.client.0.smithi102.stdout: "remote_reservations": { 2024-04-25T08:24:45.983 INFO:tasks.workunit.client.0.smithi102.stdout: "max_allowed": 1, 2024-04-25T08:24:45.983 INFO:tasks.workunit.client.0.smithi102.stdout: "min_priority": 0, 2024-04-25T08:24:45.984 INFO:tasks.workunit.client.0.smithi102.stdout: "queues": [], 2024-04-25T08:24:45.984 INFO:tasks.workunit.client.0.smithi102.stdout: "in_progress": [] 2024-04-25T08:24:45.984 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:45.984 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:24:46.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:24:46.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:24:46.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:24:46.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:24:46.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:24:46.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:24:46.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:24:46.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:24:46.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T08:24:46.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:24:46.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:24:46.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:24:47.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T08:24:47.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:24:47.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:24:47.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:298: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:24:47.465 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:24:47.465 INFO:tasks.workunit.client.0.smithi102.stdout:7.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:21:01.168856+0000 93'600 120:1148 [1,4] 1 [1,4] 1 0'0 2024-04-25T08:11:58.482060+0000 0'0 2024-04-25T08:11:58.482060+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:55:01.784864+0000 0 0 2024-04-25T08:24:47.466 INFO:tasks.workunit.client.0.smithi102.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:24:43.364194+0000 93'600 120:1180 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:11:47.100869+0000 0'0 2024-04-25T08:11:47.100869+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:37:50.476776+0000 0 0 2024-04-25T08:24:47.466 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:22:53.485751+0000 93'600 120:1197 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:11:41.048010+0000 0'0 2024-04-25T08:11:41.048010+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:29:05.681079+0000 0 0 2024-04-25T08:24:47.466 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:24:47.466 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:24:47.466 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:24:47.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:24:47.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:24:47.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:24:47.482 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:24:47.482 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:47.482 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:24:47.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.204781/ceph-osd.1.asok 2024-04-25T08:24:47.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:24:47.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.204781/ceph-osd.1.asok dump_pgstate_history 2024-04-25T08:24:47.581 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T08:24:47.581 INFO:tasks.workunit.client.0.smithi102.stdout: "pgs": [ 2024-04-25T08:24:47.581 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: "pg": "7.0", 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: "history": [ 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "62", 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:58.843230+0000", 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:58.843243+0000" 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.582 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:58.843252+0000", 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:58.994115+0000" 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:58.994117+0000", 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:58.994339+0000" 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:58.994340+0000", 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:58.994388+0000" 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:24:47.583 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:58.994389+0000", 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:59.563330+0000" 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:58.843247+0000", 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:59.563344+0000" 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:59.563695+0000", 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:59.874806+0000" 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.584 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:59.874808+0000", 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:59.874990+0000" 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:59.874991+0000", 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:00.853331+0000" 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active", 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:59.563345+0000", 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:00.853351+0000" 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.585 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:58.843244+0000", 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:00.853396+0000" 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:58.843226+0000", 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:00.853399+0000" 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "63", 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.586 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:00.853400+0000", 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:00.853601+0000" 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "92", 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:00.853604+0000", 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:00.853615+0000" 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.587 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:00.853621+0000", 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:00.972659+0000" 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:00.972661+0000", 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:00.972921+0000" 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:00.972922+0000", 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:00.972955+0000" 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.588 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:00.972956+0000", 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:01.405691+0000" 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:00.853618+0000", 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:01.405712+0000" 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:01.405982+0000", 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:01.844535+0000" 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.589 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:01.844537+0000", 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:01.844634+0000" 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:01.844634+0000", 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.717541+0000" 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active", 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:01.405713+0000", 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.717592+0000" 2024-04-25T08:24:47.590 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:00.853615+0000", 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.717714+0000" 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:00.853601+0000", 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.717724+0000" 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "92", 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.591 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.717725+0000", 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.718355+0000" 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "98", 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.718370+0000", 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.718407+0000" 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.592 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.593 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.593 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.718433+0000", 2024-04-25T08:24:47.593 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.861221+0000" 2024-04-25T08:24:47.593 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.593 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.593 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.593 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.861223+0000", 2024-04-25T08:24:47.593 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.861357+0000" 2024-04-25T08:24:47.593 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.593 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.593 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.861358+0000", 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.861385+0000" 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.861385+0000", 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:31.178893+0000" 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.718421+0000", 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:31.178914+0000" 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.594 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:31.179082+0000", 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:31.237914+0000" 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:31.237915+0000", 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:31.237991+0000" 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:31.237991+0000", 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:13.332856+0000" 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.595 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active", 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:31.178915+0000", 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:13.332913+0000" 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.718408+0000", 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:13.333128+0000" 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.718356+0000", 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:13.333136+0000" 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.596 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "98", 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:13.333137+0000", 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:13.333798+0000" 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "101", 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.597 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:13.333808+0000", 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:13.333848+0000" 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:13.333868+0000", 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:13.507725+0000" 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:13.507726+0000", 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:13.507886+0000" 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.598 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:13.333860+0000", 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:13.507908+0000" 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:13.507909+0000", 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.070112+0000" 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:13.333848+0000", 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.070173+0000" 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.599 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:13.333799+0000", 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.070183+0000" 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "101", 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.070184+0000", 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.070710+0000" 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.600 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "115", 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.070720+0000", 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.070752+0000" 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.070771+0000", 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.229851+0000" 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.601 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.229853+0000", 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.230037+0000" 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.230038+0000", 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.230074+0000" 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.230074+0000", 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.472290+0000" 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.602 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.070762+0000", 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.472310+0000" 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.475355+0000", 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.949948+0000" 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.949949+0000", 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.950169+0000" 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:24:47.603 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.950170+0000", 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:14.950649+0000" 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.950651+0000", 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:51.413971+0000" 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:51.413971+0000", 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:51.414131+0000" 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:24:47.604 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:51.414131+0000", 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:19:04.503681+0000" 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:19:04.503681+0000", 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:19:04.504401+0000" 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:19:04.504402+0000", 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:20:59.494832+0000" 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:20:59.494833+0000", 2024-04-25T08:24:47.605 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:20:59.954078+0000" 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active", 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.472312+0000", 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:20:59.954118+0000" 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.070752+0000", 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:20:59.954151+0000" 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.606 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:14.070710+0000", 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:20:59.954162+0000" 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "115", 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:20:59.954163+0000", 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:20:59.954591+0000" 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.607 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: "pg": "3.0", 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: "history": [ 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "39", 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:48.202839+0000", 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:48.202876+0000" 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.608 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:48.202902+0000", 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:48.204454+0000" 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:48.204458+0000", 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:48.204652+0000" 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:48.204653+0000", 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:48.204675+0000" 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:48.204676+0000", 2024-04-25T08:24:47.609 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:48.637518+0000" 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:48.202888+0000", 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:48.637526+0000" 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:48.637696+0000", 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:49.286767+0000" 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active", 2024-04-25T08:24:47.610 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:48.637527+0000", 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:49.286803+0000" 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:48.202877+0000", 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:49.286841+0000" 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:48.202808+0000", 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:49.286848+0000" 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.611 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "39", 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:49.286849+0000", 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:49.287220+0000" 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "90", 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:49.287227+0000", 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:49.287245+0000" 2024-04-25T08:24:47.612 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:49.287257+0000", 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:49.379473+0000" 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:49.379475+0000", 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:49.379642+0000" 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:49.379643+0000", 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:49.379669+0000" 2024-04-25T08:24:47.613 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:49.379670+0000", 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:49.987692+0000" 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:49.287252+0000", 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:49.987707+0000" 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:49.987997+0000", 2024-04-25T08:24:47.614 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:50.342688+0000" 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:50.342690+0000", 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:50.342783+0000" 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:50.342783+0000", 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.307862+0000" 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active", 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:49.987708+0000", 2024-04-25T08:24:47.615 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.307924+0000" 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:49.287246+0000", 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.308086+0000" 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:49.287221+0000", 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.308100+0000" 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.616 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "91", 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.308102+0000", 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.309014+0000" 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "107", 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.309034+0000", 2024-04-25T08:24:47.617 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.309085+0000" 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.309120+0000", 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.310256+0000" 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.310258+0000", 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.310485+0000" 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.310487+0000", 2024-04-25T08:24:47.618 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.310541+0000" 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.310542+0000", 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.719058+0000" 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.309102+0000", 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.719089+0000" 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.719302+0000", 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.909464+0000" 2024-04-25T08:24:47.619 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.909466+0000", 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:12:30.909545+0000" 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.909545+0000", 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.524461+0000" 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active", 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.719090+0000", 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.524523+0000" 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.620 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.309086+0000", 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.524733+0000" 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:12:30.309015+0000", 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.524742+0000" 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "108", 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.621 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.524743+0000", 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.525495+0000" 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "109", 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.525505+0000", 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.525537+0000" 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.622 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.525555+0000", 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.527281+0000" 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.527282+0000", 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.527419+0000" 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.525546+0000", 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.527439+0000" 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.623 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.527440+0000", 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.842238+0000" 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.525538+0000", 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.842294+0000" 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.525496+0000", 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.842305+0000" 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.624 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "109", 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.842306+0000", 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.842809+0000" 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "119", 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.625 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.842820+0000", 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.842855+0000" 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.842874+0000", 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.844272+0000" 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.844273+0000", 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.844438+0000" 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:24:47.626 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.844439+0000", 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:36.844475+0000" 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.844476+0000", 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:37.244337+0000" 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.842865+0000", 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:37.244362+0000" 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.627 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:37.247360+0000", 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:37.773293+0000" 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:37.773295+0000", 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:51.414050+0000" 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:51.414050+0000", 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:51.414468+0000" 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.628 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:51.414468+0000", 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:19:04.503500+0000" 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:19:04.503502+0000", 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:19:04.503750+0000" 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:19:04.503750+0000", 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:22:51.686982+0000" 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:24:47.629 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:22:51.686983+0000", 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:22:51.687498+0000" 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:22:51.687499+0000", 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:24:42.391753+0000" 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:24:42.391754+0000", 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:24:42.642639+0000" 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active", 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:37.244363+0000", 2024-04-25T08:24:47.630 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:24:42.642683+0000" 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.842856+0000", 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:24:42.642736+0000" 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:36.842810+0000", 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:24:42.642749+0000" 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "119", 2024-04-25T08:24:47.631 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:24:42.642751+0000", 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:24:42.643398+0000" 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: "pg": "1.0", 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: "history": [ 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.632 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "27", 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Initial", 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.043350+0000", 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:42.043592+0000" 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "27", 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.043593+0000", 2024-04-25T08:24:47.633 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:42.043668+0000" 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "29", 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.043675+0000", 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:42.043694+0000" 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.043708+0000", 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:42.317290+0000" 2024-04-25T08:24:47.634 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.317292+0000", 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:42.317432+0000" 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.317433+0000", 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:42.317456+0000" 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.317457+0000", 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:42.420251+0000" 2024-04-25T08:24:47.635 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.043701+0000", 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:42.420266+0000" 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.420684+0000", 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:42.798621+0000" 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.798623+0000", 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:42.798721+0000" 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.636 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.798722+0000", 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:43.351308+0000" 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active", 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.420267+0000", 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:43.351347+0000" 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.043695+0000", 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:43.351426+0000" 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.637 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:42.043668+0000", 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:43.351433+0000" 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "29", 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:43.351436+0000", 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:43.351862+0000" 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.638 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "103", 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:43.351870+0000", 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:43.351894+0000" 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:43.351911+0000", 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:43.352562+0000" 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.639 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:43.352564+0000", 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:43.352853+0000" 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:43.352854+0000", 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:43.352914+0000" 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:43.352915+0000", 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:43.996689+0000" 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.640 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:43.351903+0000", 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:43.996728+0000" 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:43.997303+0000", 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:44.668176+0000" 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:44.668178+0000", 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:11:44.668353+0000" 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:44.668354+0000", 2024-04-25T08:24:47.641 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:28.571349+0000" 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active", 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:43.996729+0000", 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:28.571405+0000" 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:43.351895+0000", 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:28.571616+0000" 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:11:43.351862+0000", 2024-04-25T08:24:47.642 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:28.571625+0000" 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "103", 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:28.571626+0000", 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:28.572360+0000" 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "105", 2024-04-25T08:24:47.643 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:28.572374+0000", 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:28.572406+0000" 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:28.572424+0000", 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:28.738300+0000" 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:28.738302+0000", 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:28.738470+0000" 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.644 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:28.572416+0000", 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:28.738491+0000" 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:28.738492+0000", 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:29.175025+0000" 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:28.572406+0000", 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:29.175089+0000" 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.645 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:28.572361+0000", 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:29.175099+0000" 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "105", 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.175100+0000", 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:29.175594+0000" 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.646 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "117", 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Start", 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.175604+0000", 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:29.175638+0000" 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.175657+0000", 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:29.325605+0000" 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.647 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.325608+0000", 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:29.325956+0000" 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.325958+0000", 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:29.326036+0000" 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.326037+0000", 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:29.669848+0000" 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.648 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.175648+0000", 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:29.669872+0000" 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.673085+0000", 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:18:29.959250+0000" 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.959251+0000", 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:20:59.495338+0000" 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.649 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:20:59.495339+0000", 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:20:59.496242+0000" 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:20:59.496243+0000", 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:22:51.686378+0000" 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:22:51.686380+0000", 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:22:51.994960+0000" 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.650 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary/Active", 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.669873+0000", 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:22:51.995011+0000" 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started/Primary", 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.175639+0000", 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:22:51.995063+0000" 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Started", 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:18:29.175595+0000", 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:22:51.995072+0000" 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.651 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": "117", 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: "states": [ 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "Reset", 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: "enter": "2024-04-25T08:22:51.995073+0000", 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: "exit": "2024-04-25T08:22:51.995719+0000" 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T08:24:47.652 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T08:24:47.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:302: TEST_recovery_priority: '[' 0 '!=' 0 ']' 2024-04-25T08:24:47.653 INFO:tasks.workunit.client.0.smithi102.stdout:TEST PASSED 2024-04-25T08:24:47.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:306: TEST_recovery_priority: echo TEST PASSED 2024-04-25T08:24:47.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:309: TEST_recovery_priority: delete_pool test1 2024-04-25T08:24:47.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T08:24:47.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T08:24:48.122 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test1' does not exist 2024-04-25T08:24:48.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:310: TEST_recovery_priority: delete_pool test3 2024-04-25T08:24:48.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-04-25T08:24:48.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-04-25T08:24:48.752 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test3' does not exist 2024-04-25T08:24:48.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:311: TEST_recovery_priority: delete_pool test7 2024-04-25T08:24:48.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test7 2024-04-25T08:24:48.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2024-04-25T08:24:49.498 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test7' does not exist 2024-04-25T08:24:49.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:312: TEST_recovery_priority: kill_daemons td/osd-recovery-prio 2024-04-25T08:24:49.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:24:49.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:24:49.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:24:49.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:24:49.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:24:54.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:24:54.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:313: TEST_recovery_priority: return 0 2024-04-25T08:24:54.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-04-25T08:24:54.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:24:54.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:24:54.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:24:54.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:24:54.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:24:54.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:24:54.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:24:54.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:24:54.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:24:54.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:24:54.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:24:54.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:24:54.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:24:54.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:24:54.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:24:54.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:24:54.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:24:54.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:24:54.850 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:24:54.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:24:54.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:24:54.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:24:54.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:24:54.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:54.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:24:54.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.204781 2024-04-25T08:24:54.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:24:54.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:24:54.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:24:54.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-prio 0 2024-04-25T08:24:54.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:24:54.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:24:54.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:24:54.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:24:54.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:24:54.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:24:54.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:24:54.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:24:54.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:24:54.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:24:54.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:24:54.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:24:54.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:24:54.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:24:54.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:24:54.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:24:54.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:24:54.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:24:54.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:24:54.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:24:54.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:24:54.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:24:54.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:24:54.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:54.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.204781 2024-04-25T08:24:54.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.204781 2024-04-25T08:24:54.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:24:54.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:24:54.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:24:54.947 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:24:54.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:24:54.959 INFO:tasks.workunit:Running workunit osd/osd-recovery-space.sh... 2024-04-25T08:24:54.959 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-recovery-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2024-04-25T08:24:55.010 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:24:55.014 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-space 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:21: run: local dir=td/osd-recovery-space 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:22: run: shift 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: export CEPH_MON=127.0.0.1:7221 2024-04-25T08:24:55.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: CEPH_MON=127.0.0.1:7221 2024-04-25T08:24:55.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:25: run: export CEPH_ARGS 2024-04-25T08:24:55.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: uuidgen 2024-04-25T08:24:55.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: CEPH_ARGS+='--fsid=c6ba824e-b840-43fe-93b6-97665ee7e006 --auth-supported=none ' 2024-04-25T08:24:55.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7221 ' 2024-04-25T08:24:55.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:28: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2024-04-25T08:24:55.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-04-25T08:24:55.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: export objects=600 2024-04-25T08:24:55.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: objects=600 2024-04-25T08:24:55.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: export poolprefix=test 2024-04-25T08:24:55.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: poolprefix=test 2024-04-25T08:24:55.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: set 2024-04-25T08:24:55.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:24:55.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: local funcs=TEST_recovery_test_simple 2024-04-25T08:24:55.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:34: run: for func in $funcs 2024-04-25T08:24:55.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:35: run: setup td/osd-recovery-space 2024-04-25T08:24:55.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-space 2024-04-25T08:24:55.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-space 2024-04-25T08:24:55.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:24:55.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:24:55.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:24:55.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:24:55.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:24:55.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:24:55.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:24:55.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:24:55.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:24:55.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:24:55.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:24:55.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:24:55.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:24:55.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:24:55.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:24:55.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:24:55.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:24:55.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:24:55.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:24:55.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:24:55.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:24:55.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:24:55.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:24:55.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:55.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.260802 2024-04-25T08:24:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.260802 2024-04-25T08:24:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:24:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:24:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-space 2024-04-25T08:24:55.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:24:55.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:55.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.260802 2024-04-25T08:24:55.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.260802 2024-04-25T08:24:55.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:24:55.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:24:55.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:24:55.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:24:55.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-space 1' TERM HUP INT 2024-04-25T08:24:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:36: run: TEST_recovery_test_simple td/osd-recovery-space 2024-04-25T08:24:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:89: TEST_recovery_test_simple: local dir=td/osd-recovery-space 2024-04-25T08:24:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:90: TEST_recovery_test_simple: local pools=1 2024-04-25T08:24:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:91: TEST_recovery_test_simple: local OSDS=2 2024-04-25T08:24:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:93: TEST_recovery_test_simple: run_mon td/osd-recovery-space a 2024-04-25T08:24:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-space 2024-04-25T08:24:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:24:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:24:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:24:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-space/a 2024-04-25T08:24:55.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-space/a --run-dir=td/osd-recovery-space 2024-04-25T08:24:55.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:24:55.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:24:55.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:24:55.524 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:24:55.524 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:55.524 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.260802 2024-04-25T08:24:55.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.260802/$cluster-$name.asok' 2024-04-25T08:24:55.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-space/a '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.260802/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-space/log --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:24:55.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:24:55.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:24:55.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:24:55.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:24:55.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:24:55.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:24:55.762 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:24:55.762 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:24:55.762 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:24:55.763 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:24:55.763 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:55.763 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.260802 2024-04-25T08:24:55.763 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.260802/ceph-mon.a.asok 2024-04-25T08:24:55.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:24:55.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.260802/ceph-mon.a.asok config get fsid 2024-04-25T08:24:55.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:24:55.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:24:55.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:24:55.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:24:55.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:24:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:24:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:24:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:24:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:24:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.260802 2024-04-25T08:24:55.883 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.260802/ceph-mon.a.asok 2024-04-25T08:24:55.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:24:55.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.260802/ceph-mon.a.asok config get mon_host 2024-04-25T08:24:55.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:94: TEST_recovery_test_simple: run_mgr td/osd-recovery-space x 2024-04-25T08:24:55.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-space 2024-04-25T08:24:55.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:24:55.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:24:55.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:24:55.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-space/x 2024-04-25T08:24:55.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:24:56.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:24:56.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:24:56.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:24:56.899 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:24:56.899 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:56.899 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.260802 2024-04-25T08:24:56.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.260802/$cluster-$name.asok' 2024-04-25T08:24:56.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:24:56.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-space/x '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.260802/$cluster-$name.asok' --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:24:56.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:95: TEST_recovery_test_simple: export CEPH_ARGS 2024-04-25T08:24:56.934 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: expr 2 - 1 2024-04-25T08:24:56.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: seq 0 1 2024-04-25T08:24:56.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:24:56.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 0 2024-04-25T08:24:56.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-04-25T08:24:56.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:24:56.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:24:56.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:24:56.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/0 2024-04-25T08:24:56.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c6ba824e-b840-43fe-93b6-97665ee7e006 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-04-25T08:24:56.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:24:56.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:24:56.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/0' 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/0/journal' 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.260802 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.260802/$cluster-$name.asok' 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.260802/$cluster-$name.asok' 2024-04-25T08:24:56.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/0 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=88d0c87f-6d2a-443c-9554-36fac37b6e33 2024-04-25T08:24:56.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 88d0c87f-6d2a-443c-9554-36fac37b6e33' 2024-04-25T08:24:56.942 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 88d0c87f-6d2a-443c-9554-36fac37b6e33 2024-04-25T08:24:56.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:24:56.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBYEypmeuI2ORAAWdyIrfsD4Enf9DbJzgLrzg== 2024-04-25T08:24:56.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBYEypmeuI2ORAAWdyIrfsD4Enf9DbJzgLrzg=="}' 2024-04-25T08:24:56.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 88d0c87f-6d2a-443c-9554-36fac37b6e33 -i td/osd-recovery-space/0/new.json 2024-04-25T08:24:57.304 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:24:57.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/0/new.json 2024-04-25T08:24:57.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c6ba824e-b840-43fe-93b6-97665ee7e006 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.260802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBYEypmeuI2ORAAWdyIrfsD4Enf9DbJzgLrzg== --osd-uuid 88d0c87f-6d2a-443c-9554-36fac37b6e33 2024-04-25T08:24:57.352 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:24:57.348+0000 7f85fcb847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:57.372 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:24:57.368+0000 7f85fcb847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:57.373 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:24:57.372+0000 7f85fcb847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:24:57.374 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:24:57.372+0000 7f85fcb847c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:24:57.374 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:24:57.372+0000 7f85fcb847c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:24:57.374 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:24:57.372+0000 7f85fcb847c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:24:57.412 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:24:57.408+0000 7f85fcb847c0 -1 bluestore(td/osd-recovery-space/0) _read_fsid unparsable uuid 2024-04-25T08:25:04.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/0/keyring 2024-04-25T08:25:04.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:25:04.007 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:25:04.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:25:04.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:25:04.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:25:04.556 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:25:04.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c6ba824e-b840-43fe-93b6-97665ee7e006 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.260802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:25:04.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:25:04.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:25:04.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:25:04.589 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:04.584+0000 7f79caa5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:04.591 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:04.588+0000 7f79caa5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:04.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:04.588+0000 7f79caa5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:05.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:25:05.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:25:05.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:25:05.010 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:25:05.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:25:05.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:25:05.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:05.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:25:05.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:05.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:05.146 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:05.144+0000 7f79caa5e7c0 -1 Falling back to public interface 2024-04-25T08:25:05.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:06.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:06.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:06.427 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:25:06.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:25:06.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:06.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:06.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:07.811 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:07.808+0000 7f79caa5e7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:25:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:25:07.843 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:25:07.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:07.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:08.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:09.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:09.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:09.581 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:25:09.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:25:09.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:09.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:10.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:11.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:11.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:11.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:25:11.057 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:25:11.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:11.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:11.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:12.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:12.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:12.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:25:12.509 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:25:12.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:12.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:13.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:14.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:14.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:14.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:25:14.085 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:25:14.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:14.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:14.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:15.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:15.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:15.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:25:15.508 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:25:15.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:15.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:16.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:17.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:17.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:17.061 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:25:17.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:25:17.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:17.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:17.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:18.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:18.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:18.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:25:18.494 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:25:18.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:18.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:19.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:20.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:20.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:20.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:25:20.051 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:25:20.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:20.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:20.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:21.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:21.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:21.500 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:25:21.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:25:21.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:21.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:22.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:23.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:23.470 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:25:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:25:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:23.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:24.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:25.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:25.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:25.444 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:25:25.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:25:25.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:25.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:26.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:27.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:27.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:27.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:25:27.353 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:25:27.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:27.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:28.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:29.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:29.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:29.429 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:25:29.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:25:29.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:29.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:30.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:31.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:31.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:31.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:25:31.615 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:25:31.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:31.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:32.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:33.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:33.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:33.685 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:25:33.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:25:33.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:33.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:25:34.208 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2109919478,v1:127.0.0.1:6803/2109919478] [v2:127.0.0.1:6804/2109919478,v1:127.0.0.1:6805/2109919478] exists,up 88d0c87f-6d2a-443c-9554-36fac37b6e33 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 1 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/1 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c6ba824e-b840-43fe-93b6-97665ee7e006 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-04-25T08:25:34.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/1' 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/1/journal' 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.260802 2024-04-25T08:25:34.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.260802/$cluster-$name.asok' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.260802/$cluster-$name.asok' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:25:34.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:25:34.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/1 2024-04-25T08:25:34.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:25:34.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=934f909e-fe42-4114-8ce6-36c61917b2d5 2024-04-25T08:25:34.215 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 934f909e-fe42-4114-8ce6-36c61917b2d5 2024-04-25T08:25:34.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 934f909e-fe42-4114-8ce6-36c61917b2d5' 2024-04-25T08:25:34.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:25:34.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+EypmQm5IDhAA4iaCsMfBXhZTikOdKxAbPg== 2024-04-25T08:25:34.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+EypmQm5IDhAA4iaCsMfBXhZTikOdKxAbPg=="}' 2024-04-25T08:25:34.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 934f909e-fe42-4114-8ce6-36c61917b2d5 -i td/osd-recovery-space/1/new.json 2024-04-25T08:25:34.762 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:25:34.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/1/new.json 2024-04-25T08:25:34.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c6ba824e-b840-43fe-93b6-97665ee7e006 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.260802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+EypmQm5IDhAA4iaCsMfBXhZTikOdKxAbPg== --osd-uuid 934f909e-fe42-4114-8ce6-36c61917b2d5 2024-04-25T08:25:34.815 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:34.815+0000 7fd760bd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:34.876 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:34.875+0000 7fd760bd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:34.878 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:34.879+0000 7fd760bd77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:34.878 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:34.879+0000 7fd760bd77c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:25:34.878 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:34.879+0000 7fd760bd77c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:25:34.878 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:34.879+0000 7fd760bd77c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:25:35.003 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:35.003+0000 7fd760bd77c0 -1 bluestore(td/osd-recovery-space/1) _read_fsid unparsable uuid 2024-04-25T08:25:42.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/1/keyring 2024-04-25T08:25:42.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:25:42.328 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T08:25:42.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:25:42.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:25:42.884 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:25:42.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:25:42.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:25:42.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:25:42.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c6ba824e-b840-43fe-93b6-97665ee7e006 --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.260802/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:25:42.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:25:42.918 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:42.919+0000 7ff44ce077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:42.920 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:42.919+0000 7ff44ce077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:42.921 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:42.923+0000 7ff44ce077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:25:43.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:25:43.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:25:43.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:25:43.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:25:43.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:25:43.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:43.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:25:43.339 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:25:43.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:43.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:25:43.487 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:43.487+0000 7ff44ce077c0 -1 Falling back to public interface 2024-04-25T08:25:43.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:44.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:44.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:44.798 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:25:44.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:25:44.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:44.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:25:45.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:46.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:46.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:46.202 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:25:46.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:25:46.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:46.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:25:46.221 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:25:46.223+0000 7ff44ce077c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:25:46.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:47.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:47.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:47.629 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:25:47.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:25:47.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:47.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:25:48.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:49.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:49.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:49.084 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:25:49.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:25:49.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:49.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:25:49.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:50.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:50.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:50.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:25:50.608 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:25:50.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:50.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:25:51.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:52.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:52.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:52.070 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:25:52.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:25:52.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:52.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:25:52.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:53.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:53.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:53.652 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:25:53.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:25:53.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:53.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:25:54.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:55.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:55.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:55.086 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:25:55.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:25:55.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:55.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:25:55.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:56.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:56.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:56.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:25:56.629 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:25:56.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:56.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:25:57.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:58.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:58.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:58.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:25:58.068 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:25:58.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:58.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:25:58.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:25:59.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:25:59.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:25:59.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:25:59.603 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:25:59.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:25:59.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:00.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:01.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:01.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:01.088 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:26:01.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:26:01.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:01.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:01.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:02.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:02.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:02.583 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:26:02.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:26:02.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:02.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:03.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:04.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:04.478 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:26:04.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:04.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:26:04.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:04.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:05.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:06.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:06.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:06.283 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:26:06.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:26:06.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:06.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:07.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:08.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:08.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:08.407 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:26:08.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:26:08.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:08.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:09.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:10.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:10.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:10.224 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:26:10.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:26:10.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:10.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:10.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:11.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:11.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:11.771 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:26:11.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:26:11.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:11.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:12.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:26:13.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:26:13.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:26:13.718 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:26:13.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:26:13.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:26:13.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:26:14.167 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2694665599,v1:127.0.0.1:6811/2694665599] [v2:127.0.0.1:6812/2694665599,v1:127.0.0.1:6813/2694665599] exists,up 934f909e-fe42-4114-8ce6-36c61917b2d5 2024-04-25T08:26:14.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:26:14.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:26:14.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:26:14.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:102: TEST_recovery_test_simple: ceph osd set-nearfull-ratio .40 2024-04-25T08:26:14.817 INFO:tasks.workunit.client.0.smithi102.stderr:osd set-nearfull-ratio 0.4 2024-04-25T08:26:14.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: ceph osd set-backfillfull-ratio .45 2024-04-25T08:26:15.787 INFO:tasks.workunit.client.0.smithi102.stderr:osd set-backfillfull-ratio 0.45 2024-04-25T08:26:15.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:104: TEST_recovery_test_simple: ceph osd set-full-ratio .50 2024-04-25T08:26:16.624 INFO:tasks.workunit.client.0.smithi102.stderr:osd set-full-ratio 0.5 2024-04-25T08:26:16.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:26:16.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-04-25T08:26:16.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:108: TEST_recovery_test_simple: create_pool test1 1 1 2024-04-25T08:26:16.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T08:26:17.448 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test1' already exists 2024-04-25T08:26:17.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:26:18.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:109: TEST_recovery_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-04-25T08:26:19.494 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 1 2024-04-25T08:26:19.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:112: TEST_recovery_test_simple: wait_for_clean 2024-04-25T08:26:19.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:26:19.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:26:19.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:26:19.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:26:19.520 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:26:19.520 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:26:19.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:26:19.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:26:19.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:26:19.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:26:19.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:26:19.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:26:19.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:26:19.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:26:19.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:26:20.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:26:20.133 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-04-25T08:26:20.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:26:20.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:20.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:26:20.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836496 2024-04-25T08:26:20.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836496 2024-04-25T08:26:20.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836496' 2024-04-25T08:26:20.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:20.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:26:20.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672969 2024-04-25T08:26:20.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672969 2024-04-25T08:26:20.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836496 1-42949672969' 2024-04-25T08:26:20.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:20.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836496 2024-04-25T08:26:20.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:20.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:26:20.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836496 2024-04-25T08:26:20.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:20.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836496 2024-04-25T08:26:20.546 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836496 2024-04-25T08:26:20.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2024-04-25T08:26:20.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:20.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836494 -lt 21474836496 2024-04-25T08:26:20.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:26:21.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:26:21.940 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:22.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836496 -lt 21474836496 2024-04-25T08:26:22.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:22.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672969 2024-04-25T08:26:22.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:22.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:26:22.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672969 2024-04-25T08:26:22.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:22.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672969 2024-04-25T08:26:22.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-04-25T08:26:22.399 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672969 2024-04-25T08:26:22.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:26:22.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672969 -lt 42949672969 2024-04-25T08:26:22.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:26:22.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:26:22.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:26:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:26:23.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:26:23.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:26:23.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:26:23.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:26:23.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:26:23.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:26:23.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:26:23.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:26:23.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:26:23.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:26:23.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:26:24.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:26:24.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:26:24.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:26:24.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: dd if=/dev/urandom of=td/osd-recovery-space/datafile bs=1024 count=5 2024-04-25T08:26:24.402 INFO:tasks.workunit.client.0.smithi102.stderr:5+0 records in 2024-04-25T08:26:24.402 INFO:tasks.workunit.client.0.smithi102.stderr:5+0 records out 2024-04-25T08:26:24.402 INFO:tasks.workunit.client.0.smithi102.stderr:5120 bytes (5.1 kB, 5.0 KiB) copied, 0.000101864 s, 50.3 MB/s 2024-04-25T08:26:24.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: seq 1 600 2024-04-25T08:26:24.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:24.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj1 td/osd-recovery-space/datafile 2024-04-25T08:26:24.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:24.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj2 td/osd-recovery-space/datafile 2024-04-25T08:26:24.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:24.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj3 td/osd-recovery-space/datafile 2024-04-25T08:26:24.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:24.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj4 td/osd-recovery-space/datafile 2024-04-25T08:26:24.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:24.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj5 td/osd-recovery-space/datafile 2024-04-25T08:26:25.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:25.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj6 td/osd-recovery-space/datafile 2024-04-25T08:26:25.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:25.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj7 td/osd-recovery-space/datafile 2024-04-25T08:26:25.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:25.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj8 td/osd-recovery-space/datafile 2024-04-25T08:26:25.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:25.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj9 td/osd-recovery-space/datafile 2024-04-25T08:26:25.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:25.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj10 td/osd-recovery-space/datafile 2024-04-25T08:26:25.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:25.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj11 td/osd-recovery-space/datafile 2024-04-25T08:26:25.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:25.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj12 td/osd-recovery-space/datafile 2024-04-25T08:26:25.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:25.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj13 td/osd-recovery-space/datafile 2024-04-25T08:26:25.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:25.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj14 td/osd-recovery-space/datafile 2024-04-25T08:26:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:25.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj15 td/osd-recovery-space/datafile 2024-04-25T08:26:26.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:26.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj16 td/osd-recovery-space/datafile 2024-04-25T08:26:26.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:26.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj17 td/osd-recovery-space/datafile 2024-04-25T08:26:26.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:26.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj18 td/osd-recovery-space/datafile 2024-04-25T08:26:26.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:26.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj19 td/osd-recovery-space/datafile 2024-04-25T08:26:26.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:26.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj20 td/osd-recovery-space/datafile 2024-04-25T08:26:26.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:26.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj21 td/osd-recovery-space/datafile 2024-04-25T08:26:26.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:26.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj22 td/osd-recovery-space/datafile 2024-04-25T08:26:26.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:26.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj23 td/osd-recovery-space/datafile 2024-04-25T08:26:27.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:27.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj24 td/osd-recovery-space/datafile 2024-04-25T08:26:27.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:27.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj25 td/osd-recovery-space/datafile 2024-04-25T08:26:27.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:27.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj26 td/osd-recovery-space/datafile 2024-04-25T08:26:27.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:27.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj27 td/osd-recovery-space/datafile 2024-04-25T08:26:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj28 td/osd-recovery-space/datafile 2024-04-25T08:26:27.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:27.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj29 td/osd-recovery-space/datafile 2024-04-25T08:26:27.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:27.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj30 td/osd-recovery-space/datafile 2024-04-25T08:26:27.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:27.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj31 td/osd-recovery-space/datafile 2024-04-25T08:26:27.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:27.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj32 td/osd-recovery-space/datafile 2024-04-25T08:26:28.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:28.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj33 td/osd-recovery-space/datafile 2024-04-25T08:26:28.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:28.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj34 td/osd-recovery-space/datafile 2024-04-25T08:26:28.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:28.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj35 td/osd-recovery-space/datafile 2024-04-25T08:26:28.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:28.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj36 td/osd-recovery-space/datafile 2024-04-25T08:26:28.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:28.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj37 td/osd-recovery-space/datafile 2024-04-25T08:26:28.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:28.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj38 td/osd-recovery-space/datafile 2024-04-25T08:26:28.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:28.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj39 td/osd-recovery-space/datafile 2024-04-25T08:26:28.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:28.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj40 td/osd-recovery-space/datafile 2024-04-25T08:26:28.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:28.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj41 td/osd-recovery-space/datafile 2024-04-25T08:26:29.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:29.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj42 td/osd-recovery-space/datafile 2024-04-25T08:26:29.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:29.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj43 td/osd-recovery-space/datafile 2024-04-25T08:26:29.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:29.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj44 td/osd-recovery-space/datafile 2024-04-25T08:26:29.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:29.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj45 td/osd-recovery-space/datafile 2024-04-25T08:26:29.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:29.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj46 td/osd-recovery-space/datafile 2024-04-25T08:26:29.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:29.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj47 td/osd-recovery-space/datafile 2024-04-25T08:26:29.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:29.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj48 td/osd-recovery-space/datafile 2024-04-25T08:26:29.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:29.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj49 td/osd-recovery-space/datafile 2024-04-25T08:26:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:29.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj50 td/osd-recovery-space/datafile 2024-04-25T08:26:30.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:30.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj51 td/osd-recovery-space/datafile 2024-04-25T08:26:30.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:30.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj52 td/osd-recovery-space/datafile 2024-04-25T08:26:30.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:30.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj53 td/osd-recovery-space/datafile 2024-04-25T08:26:30.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:30.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj54 td/osd-recovery-space/datafile 2024-04-25T08:26:30.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:30.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj55 td/osd-recovery-space/datafile 2024-04-25T08:26:30.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:30.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj56 td/osd-recovery-space/datafile 2024-04-25T08:26:30.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:30.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj57 td/osd-recovery-space/datafile 2024-04-25T08:26:30.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:30.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj58 td/osd-recovery-space/datafile 2024-04-25T08:26:30.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:30.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj59 td/osd-recovery-space/datafile 2024-04-25T08:26:31.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:31.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj60 td/osd-recovery-space/datafile 2024-04-25T08:26:31.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:31.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj61 td/osd-recovery-space/datafile 2024-04-25T08:26:31.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:31.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj62 td/osd-recovery-space/datafile 2024-04-25T08:26:31.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:31.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj63 td/osd-recovery-space/datafile 2024-04-25T08:26:31.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:31.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj64 td/osd-recovery-space/datafile 2024-04-25T08:26:31.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:31.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj65 td/osd-recovery-space/datafile 2024-04-25T08:26:31.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:31.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj66 td/osd-recovery-space/datafile 2024-04-25T08:26:31.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:31.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj67 td/osd-recovery-space/datafile 2024-04-25T08:26:31.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:31.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj68 td/osd-recovery-space/datafile 2024-04-25T08:26:32.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:32.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj69 td/osd-recovery-space/datafile 2024-04-25T08:26:32.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:32.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj70 td/osd-recovery-space/datafile 2024-04-25T08:26:32.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:32.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj71 td/osd-recovery-space/datafile 2024-04-25T08:26:32.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:32.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj72 td/osd-recovery-space/datafile 2024-04-25T08:26:32.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:32.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj73 td/osd-recovery-space/datafile 2024-04-25T08:26:32.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:32.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj74 td/osd-recovery-space/datafile 2024-04-25T08:26:32.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:32.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj75 td/osd-recovery-space/datafile 2024-04-25T08:26:32.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:32.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj76 td/osd-recovery-space/datafile 2024-04-25T08:26:32.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:32.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj77 td/osd-recovery-space/datafile 2024-04-25T08:26:33.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:33.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj78 td/osd-recovery-space/datafile 2024-04-25T08:26:33.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:33.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj79 td/osd-recovery-space/datafile 2024-04-25T08:26:33.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:33.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj80 td/osd-recovery-space/datafile 2024-04-25T08:26:33.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:33.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj81 td/osd-recovery-space/datafile 2024-04-25T08:26:33.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:33.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj82 td/osd-recovery-space/datafile 2024-04-25T08:26:33.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:33.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj83 td/osd-recovery-space/datafile 2024-04-25T08:26:33.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:33.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj84 td/osd-recovery-space/datafile 2024-04-25T08:26:33.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:33.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj85 td/osd-recovery-space/datafile 2024-04-25T08:26:33.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:33.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj86 td/osd-recovery-space/datafile 2024-04-25T08:26:34.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:34.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj87 td/osd-recovery-space/datafile 2024-04-25T08:26:34.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:34.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj88 td/osd-recovery-space/datafile 2024-04-25T08:26:34.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:34.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj89 td/osd-recovery-space/datafile 2024-04-25T08:26:34.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:34.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj90 td/osd-recovery-space/datafile 2024-04-25T08:26:34.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:34.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj91 td/osd-recovery-space/datafile 2024-04-25T08:26:34.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:34.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj92 td/osd-recovery-space/datafile 2024-04-25T08:26:34.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:34.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj93 td/osd-recovery-space/datafile 2024-04-25T08:26:34.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:34.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj94 td/osd-recovery-space/datafile 2024-04-25T08:26:34.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:34.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj95 td/osd-recovery-space/datafile 2024-04-25T08:26:35.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:35.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj96 td/osd-recovery-space/datafile 2024-04-25T08:26:35.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:35.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj97 td/osd-recovery-space/datafile 2024-04-25T08:26:35.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:35.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj98 td/osd-recovery-space/datafile 2024-04-25T08:26:35.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:35.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj99 td/osd-recovery-space/datafile 2024-04-25T08:26:35.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:35.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj100 td/osd-recovery-space/datafile 2024-04-25T08:26:35.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:35.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj101 td/osd-recovery-space/datafile 2024-04-25T08:26:35.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:35.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj102 td/osd-recovery-space/datafile 2024-04-25T08:26:35.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:35.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj103 td/osd-recovery-space/datafile 2024-04-25T08:26:35.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:35.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj104 td/osd-recovery-space/datafile 2024-04-25T08:26:36.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:36.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj105 td/osd-recovery-space/datafile 2024-04-25T08:26:36.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:36.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj106 td/osd-recovery-space/datafile 2024-04-25T08:26:36.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:36.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj107 td/osd-recovery-space/datafile 2024-04-25T08:26:36.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:36.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj108 td/osd-recovery-space/datafile 2024-04-25T08:26:36.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:36.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj109 td/osd-recovery-space/datafile 2024-04-25T08:26:36.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:36.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj110 td/osd-recovery-space/datafile 2024-04-25T08:26:36.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:36.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj111 td/osd-recovery-space/datafile 2024-04-25T08:26:36.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:36.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj112 td/osd-recovery-space/datafile 2024-04-25T08:26:37.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:37.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj113 td/osd-recovery-space/datafile 2024-04-25T08:26:37.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:37.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj114 td/osd-recovery-space/datafile 2024-04-25T08:26:37.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:37.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj115 td/osd-recovery-space/datafile 2024-04-25T08:26:37.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:37.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj116 td/osd-recovery-space/datafile 2024-04-25T08:26:37.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:37.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj117 td/osd-recovery-space/datafile 2024-04-25T08:26:37.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:37.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj118 td/osd-recovery-space/datafile 2024-04-25T08:26:37.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:37.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj119 td/osd-recovery-space/datafile 2024-04-25T08:26:37.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:37.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj120 td/osd-recovery-space/datafile 2024-04-25T08:26:37.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:37.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj121 td/osd-recovery-space/datafile 2024-04-25T08:26:38.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:38.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj122 td/osd-recovery-space/datafile 2024-04-25T08:26:38.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:38.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj123 td/osd-recovery-space/datafile 2024-04-25T08:26:38.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:38.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj124 td/osd-recovery-space/datafile 2024-04-25T08:26:38.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:38.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj125 td/osd-recovery-space/datafile 2024-04-25T08:26:38.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:38.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj126 td/osd-recovery-space/datafile 2024-04-25T08:26:38.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:38.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj127 td/osd-recovery-space/datafile 2024-04-25T08:26:38.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:38.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj128 td/osd-recovery-space/datafile 2024-04-25T08:26:38.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:38.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj129 td/osd-recovery-space/datafile 2024-04-25T08:26:38.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:38.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj130 td/osd-recovery-space/datafile 2024-04-25T08:26:39.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:39.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj131 td/osd-recovery-space/datafile 2024-04-25T08:26:39.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:39.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj132 td/osd-recovery-space/datafile 2024-04-25T08:26:39.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:39.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj133 td/osd-recovery-space/datafile 2024-04-25T08:26:39.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:39.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj134 td/osd-recovery-space/datafile 2024-04-25T08:26:39.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:39.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj135 td/osd-recovery-space/datafile 2024-04-25T08:26:39.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:39.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj136 td/osd-recovery-space/datafile 2024-04-25T08:26:39.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:39.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj137 td/osd-recovery-space/datafile 2024-04-25T08:26:39.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:39.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj138 td/osd-recovery-space/datafile 2024-04-25T08:26:39.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:39.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj139 td/osd-recovery-space/datafile 2024-04-25T08:26:40.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:40.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj140 td/osd-recovery-space/datafile 2024-04-25T08:26:40.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:40.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj141 td/osd-recovery-space/datafile 2024-04-25T08:26:40.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:40.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj142 td/osd-recovery-space/datafile 2024-04-25T08:26:40.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:40.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj143 td/osd-recovery-space/datafile 2024-04-25T08:26:40.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:40.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj144 td/osd-recovery-space/datafile 2024-04-25T08:26:40.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:40.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj145 td/osd-recovery-space/datafile 2024-04-25T08:26:40.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:40.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj146 td/osd-recovery-space/datafile 2024-04-25T08:26:40.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:40.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj147 td/osd-recovery-space/datafile 2024-04-25T08:26:40.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:40.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj148 td/osd-recovery-space/datafile 2024-04-25T08:26:41.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:41.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj149 td/osd-recovery-space/datafile 2024-04-25T08:26:41.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:41.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj150 td/osd-recovery-space/datafile 2024-04-25T08:26:41.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:41.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj151 td/osd-recovery-space/datafile 2024-04-25T08:26:41.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:41.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj152 td/osd-recovery-space/datafile 2024-04-25T08:26:41.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:41.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj153 td/osd-recovery-space/datafile 2024-04-25T08:26:41.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:41.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj154 td/osd-recovery-space/datafile 2024-04-25T08:26:41.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:41.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj155 td/osd-recovery-space/datafile 2024-04-25T08:26:41.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:41.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj156 td/osd-recovery-space/datafile 2024-04-25T08:26:41.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:41.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj157 td/osd-recovery-space/datafile 2024-04-25T08:26:42.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:42.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj158 td/osd-recovery-space/datafile 2024-04-25T08:26:42.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:42.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj159 td/osd-recovery-space/datafile 2024-04-25T08:26:42.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:42.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj160 td/osd-recovery-space/datafile 2024-04-25T08:26:42.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:42.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj161 td/osd-recovery-space/datafile 2024-04-25T08:26:42.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:42.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj162 td/osd-recovery-space/datafile 2024-04-25T08:26:42.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:42.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj163 td/osd-recovery-space/datafile 2024-04-25T08:26:42.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:42.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj164 td/osd-recovery-space/datafile 2024-04-25T08:26:42.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:42.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj165 td/osd-recovery-space/datafile 2024-04-25T08:26:42.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:42.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj166 td/osd-recovery-space/datafile 2024-04-25T08:26:43.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:43.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj167 td/osd-recovery-space/datafile 2024-04-25T08:26:43.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:43.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj168 td/osd-recovery-space/datafile 2024-04-25T08:26:43.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:43.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj169 td/osd-recovery-space/datafile 2024-04-25T08:26:43.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:43.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj170 td/osd-recovery-space/datafile 2024-04-25T08:26:43.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:43.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj171 td/osd-recovery-space/datafile 2024-04-25T08:26:43.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:43.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj172 td/osd-recovery-space/datafile 2024-04-25T08:26:43.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:43.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj173 td/osd-recovery-space/datafile 2024-04-25T08:26:43.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:43.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj174 td/osd-recovery-space/datafile 2024-04-25T08:26:43.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:43.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj175 td/osd-recovery-space/datafile 2024-04-25T08:26:43.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:43.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj176 td/osd-recovery-space/datafile 2024-04-25T08:26:44.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:44.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj177 td/osd-recovery-space/datafile 2024-04-25T08:26:44.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:44.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj178 td/osd-recovery-space/datafile 2024-04-25T08:26:44.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:44.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj179 td/osd-recovery-space/datafile 2024-04-25T08:26:44.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:44.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj180 td/osd-recovery-space/datafile 2024-04-25T08:26:44.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:44.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj181 td/osd-recovery-space/datafile 2024-04-25T08:26:44.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:44.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj182 td/osd-recovery-space/datafile 2024-04-25T08:26:44.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:44.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj183 td/osd-recovery-space/datafile 2024-04-25T08:26:44.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:44.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj184 td/osd-recovery-space/datafile 2024-04-25T08:26:45.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:45.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj185 td/osd-recovery-space/datafile 2024-04-25T08:26:45.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:45.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj186 td/osd-recovery-space/datafile 2024-04-25T08:26:45.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:45.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj187 td/osd-recovery-space/datafile 2024-04-25T08:26:45.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:45.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj188 td/osd-recovery-space/datafile 2024-04-25T08:26:45.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:45.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj189 td/osd-recovery-space/datafile 2024-04-25T08:26:45.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:45.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj190 td/osd-recovery-space/datafile 2024-04-25T08:26:45.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:45.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj191 td/osd-recovery-space/datafile 2024-04-25T08:26:45.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:45.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj192 td/osd-recovery-space/datafile 2024-04-25T08:26:45.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:45.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj193 td/osd-recovery-space/datafile 2024-04-25T08:26:45.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:45.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj194 td/osd-recovery-space/datafile 2024-04-25T08:26:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:46.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj195 td/osd-recovery-space/datafile 2024-04-25T08:26:46.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:46.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj196 td/osd-recovery-space/datafile 2024-04-25T08:26:46.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:46.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj197 td/osd-recovery-space/datafile 2024-04-25T08:26:46.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:46.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj198 td/osd-recovery-space/datafile 2024-04-25T08:26:46.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:46.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj199 td/osd-recovery-space/datafile 2024-04-25T08:26:46.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:46.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj200 td/osd-recovery-space/datafile 2024-04-25T08:26:46.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:46.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj201 td/osd-recovery-space/datafile 2024-04-25T08:26:46.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:46.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj202 td/osd-recovery-space/datafile 2024-04-25T08:26:47.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:47.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj203 td/osd-recovery-space/datafile 2024-04-25T08:26:47.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:47.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj204 td/osd-recovery-space/datafile 2024-04-25T08:26:47.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:47.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj205 td/osd-recovery-space/datafile 2024-04-25T08:26:47.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:47.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj206 td/osd-recovery-space/datafile 2024-04-25T08:26:47.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:47.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj207 td/osd-recovery-space/datafile 2024-04-25T08:26:47.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:47.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj208 td/osd-recovery-space/datafile 2024-04-25T08:26:47.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:47.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj209 td/osd-recovery-space/datafile 2024-04-25T08:26:47.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:47.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj210 td/osd-recovery-space/datafile 2024-04-25T08:26:47.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:47.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj211 td/osd-recovery-space/datafile 2024-04-25T08:26:48.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:48.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj212 td/osd-recovery-space/datafile 2024-04-25T08:26:48.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:48.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj213 td/osd-recovery-space/datafile 2024-04-25T08:26:48.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:48.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj214 td/osd-recovery-space/datafile 2024-04-25T08:26:48.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:48.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj215 td/osd-recovery-space/datafile 2024-04-25T08:26:48.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:48.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj216 td/osd-recovery-space/datafile 2024-04-25T08:26:48.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:48.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj217 td/osd-recovery-space/datafile 2024-04-25T08:26:48.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:48.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj218 td/osd-recovery-space/datafile 2024-04-25T08:26:48.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:48.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj219 td/osd-recovery-space/datafile 2024-04-25T08:26:48.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:48.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj220 td/osd-recovery-space/datafile 2024-04-25T08:26:49.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:49.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj221 td/osd-recovery-space/datafile 2024-04-25T08:26:49.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:49.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj222 td/osd-recovery-space/datafile 2024-04-25T08:26:49.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:49.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj223 td/osd-recovery-space/datafile 2024-04-25T08:26:49.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:49.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj224 td/osd-recovery-space/datafile 2024-04-25T08:26:49.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:49.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj225 td/osd-recovery-space/datafile 2024-04-25T08:26:49.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:49.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj226 td/osd-recovery-space/datafile 2024-04-25T08:26:49.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:49.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj227 td/osd-recovery-space/datafile 2024-04-25T08:26:49.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:49.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj228 td/osd-recovery-space/datafile 2024-04-25T08:26:49.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:49.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj229 td/osd-recovery-space/datafile 2024-04-25T08:26:50.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:50.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj230 td/osd-recovery-space/datafile 2024-04-25T08:26:50.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:50.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj231 td/osd-recovery-space/datafile 2024-04-25T08:26:50.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:50.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj232 td/osd-recovery-space/datafile 2024-04-25T08:26:50.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:50.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj233 td/osd-recovery-space/datafile 2024-04-25T08:26:50.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:50.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj234 td/osd-recovery-space/datafile 2024-04-25T08:26:50.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:50.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj235 td/osd-recovery-space/datafile 2024-04-25T08:26:50.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:50.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj236 td/osd-recovery-space/datafile 2024-04-25T08:26:50.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:50.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj237 td/osd-recovery-space/datafile 2024-04-25T08:26:50.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:50.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj238 td/osd-recovery-space/datafile 2024-04-25T08:26:51.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:51.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj239 td/osd-recovery-space/datafile 2024-04-25T08:26:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:51.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj240 td/osd-recovery-space/datafile 2024-04-25T08:26:51.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:51.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj241 td/osd-recovery-space/datafile 2024-04-25T08:26:51.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:51.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj242 td/osd-recovery-space/datafile 2024-04-25T08:26:51.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:51.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj243 td/osd-recovery-space/datafile 2024-04-25T08:26:51.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:51.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj244 td/osd-recovery-space/datafile 2024-04-25T08:26:51.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:51.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj245 td/osd-recovery-space/datafile 2024-04-25T08:26:51.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:51.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj246 td/osd-recovery-space/datafile 2024-04-25T08:26:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj247 td/osd-recovery-space/datafile 2024-04-25T08:26:52.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj248 td/osd-recovery-space/datafile 2024-04-25T08:26:52.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:52.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj249 td/osd-recovery-space/datafile 2024-04-25T08:26:52.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:52.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj250 td/osd-recovery-space/datafile 2024-04-25T08:26:52.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:52.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj251 td/osd-recovery-space/datafile 2024-04-25T08:26:52.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:52.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj252 td/osd-recovery-space/datafile 2024-04-25T08:26:52.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:52.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj253 td/osd-recovery-space/datafile 2024-04-25T08:26:52.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:52.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj254 td/osd-recovery-space/datafile 2024-04-25T08:26:52.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:52.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj255 td/osd-recovery-space/datafile 2024-04-25T08:26:53.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:53.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj256 td/osd-recovery-space/datafile 2024-04-25T08:26:53.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:53.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj257 td/osd-recovery-space/datafile 2024-04-25T08:26:53.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:53.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj258 td/osd-recovery-space/datafile 2024-04-25T08:26:53.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:53.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj259 td/osd-recovery-space/datafile 2024-04-25T08:26:53.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:53.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj260 td/osd-recovery-space/datafile 2024-04-25T08:26:53.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:53.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj261 td/osd-recovery-space/datafile 2024-04-25T08:26:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:53.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj262 td/osd-recovery-space/datafile 2024-04-25T08:26:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:53.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj263 td/osd-recovery-space/datafile 2024-04-25T08:26:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:53.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj264 td/osd-recovery-space/datafile 2024-04-25T08:26:53.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:53.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj265 td/osd-recovery-space/datafile 2024-04-25T08:26:54.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:54.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj266 td/osd-recovery-space/datafile 2024-04-25T08:26:54.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:54.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj267 td/osd-recovery-space/datafile 2024-04-25T08:26:54.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:54.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj268 td/osd-recovery-space/datafile 2024-04-25T08:26:54.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:54.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj269 td/osd-recovery-space/datafile 2024-04-25T08:26:55.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:55.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj270 td/osd-recovery-space/datafile 2024-04-25T08:26:55.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:55.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj271 td/osd-recovery-space/datafile 2024-04-25T08:26:55.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:55.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj272 td/osd-recovery-space/datafile 2024-04-25T08:26:55.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:55.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj273 td/osd-recovery-space/datafile 2024-04-25T08:26:55.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:55.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj274 td/osd-recovery-space/datafile 2024-04-25T08:26:55.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:55.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj275 td/osd-recovery-space/datafile 2024-04-25T08:26:55.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:55.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj276 td/osd-recovery-space/datafile 2024-04-25T08:26:55.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:55.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj277 td/osd-recovery-space/datafile 2024-04-25T08:26:55.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:55.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj278 td/osd-recovery-space/datafile 2024-04-25T08:26:56.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:56.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj279 td/osd-recovery-space/datafile 2024-04-25T08:26:56.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:56.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj280 td/osd-recovery-space/datafile 2024-04-25T08:26:56.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:56.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj281 td/osd-recovery-space/datafile 2024-04-25T08:26:56.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:56.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj282 td/osd-recovery-space/datafile 2024-04-25T08:26:56.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:56.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj283 td/osd-recovery-space/datafile 2024-04-25T08:26:56.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:56.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj284 td/osd-recovery-space/datafile 2024-04-25T08:26:56.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:56.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj285 td/osd-recovery-space/datafile 2024-04-25T08:26:56.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:56.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj286 td/osd-recovery-space/datafile 2024-04-25T08:26:56.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:56.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj287 td/osd-recovery-space/datafile 2024-04-25T08:26:57.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:57.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj288 td/osd-recovery-space/datafile 2024-04-25T08:26:57.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:57.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj289 td/osd-recovery-space/datafile 2024-04-25T08:26:57.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:57.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj290 td/osd-recovery-space/datafile 2024-04-25T08:26:57.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:57.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj291 td/osd-recovery-space/datafile 2024-04-25T08:26:57.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:57.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj292 td/osd-recovery-space/datafile 2024-04-25T08:26:57.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:57.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj293 td/osd-recovery-space/datafile 2024-04-25T08:26:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:57.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj294 td/osd-recovery-space/datafile 2024-04-25T08:26:57.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:57.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj295 td/osd-recovery-space/datafile 2024-04-25T08:26:57.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:57.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj296 td/osd-recovery-space/datafile 2024-04-25T08:26:58.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:58.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj297 td/osd-recovery-space/datafile 2024-04-25T08:26:58.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:58.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj298 td/osd-recovery-space/datafile 2024-04-25T08:26:58.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:58.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj299 td/osd-recovery-space/datafile 2024-04-25T08:26:58.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:58.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj300 td/osd-recovery-space/datafile 2024-04-25T08:26:58.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:58.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj301 td/osd-recovery-space/datafile 2024-04-25T08:26:58.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:58.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj302 td/osd-recovery-space/datafile 2024-04-25T08:26:58.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:58.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj303 td/osd-recovery-space/datafile 2024-04-25T08:26:58.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:58.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj304 td/osd-recovery-space/datafile 2024-04-25T08:26:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj305 td/osd-recovery-space/datafile 2024-04-25T08:26:59.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:59.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj306 td/osd-recovery-space/datafile 2024-04-25T08:26:59.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:59.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj307 td/osd-recovery-space/datafile 2024-04-25T08:26:59.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:59.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj308 td/osd-recovery-space/datafile 2024-04-25T08:26:59.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:59.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj309 td/osd-recovery-space/datafile 2024-04-25T08:26:59.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:59.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj310 td/osd-recovery-space/datafile 2024-04-25T08:26:59.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:59.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj311 td/osd-recovery-space/datafile 2024-04-25T08:26:59.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:59.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj312 td/osd-recovery-space/datafile 2024-04-25T08:26:59.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:59.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj313 td/osd-recovery-space/datafile 2024-04-25T08:26:59.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:26:59.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj314 td/osd-recovery-space/datafile 2024-04-25T08:27:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj315 td/osd-recovery-space/datafile 2024-04-25T08:27:00.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:00.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj316 td/osd-recovery-space/datafile 2024-04-25T08:27:00.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:00.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj317 td/osd-recovery-space/datafile 2024-04-25T08:27:00.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:00.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj318 td/osd-recovery-space/datafile 2024-04-25T08:27:00.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:00.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj319 td/osd-recovery-space/datafile 2024-04-25T08:27:00.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:00.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj320 td/osd-recovery-space/datafile 2024-04-25T08:27:00.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:00.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj321 td/osd-recovery-space/datafile 2024-04-25T08:27:00.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:00.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj322 td/osd-recovery-space/datafile 2024-04-25T08:27:01.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj323 td/osd-recovery-space/datafile 2024-04-25T08:27:01.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj324 td/osd-recovery-space/datafile 2024-04-25T08:27:01.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj325 td/osd-recovery-space/datafile 2024-04-25T08:27:01.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj326 td/osd-recovery-space/datafile 2024-04-25T08:27:01.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj327 td/osd-recovery-space/datafile 2024-04-25T08:27:01.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj328 td/osd-recovery-space/datafile 2024-04-25T08:27:01.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj329 td/osd-recovery-space/datafile 2024-04-25T08:27:01.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj330 td/osd-recovery-space/datafile 2024-04-25T08:27:01.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj331 td/osd-recovery-space/datafile 2024-04-25T08:27:01.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj332 td/osd-recovery-space/datafile 2024-04-25T08:27:01.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj333 td/osd-recovery-space/datafile 2024-04-25T08:27:01.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj334 td/osd-recovery-space/datafile 2024-04-25T08:27:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj335 td/osd-recovery-space/datafile 2024-04-25T08:27:02.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:02.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj336 td/osd-recovery-space/datafile 2024-04-25T08:27:02.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:02.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj337 td/osd-recovery-space/datafile 2024-04-25T08:27:02.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:02.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj338 td/osd-recovery-space/datafile 2024-04-25T08:27:02.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:02.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj339 td/osd-recovery-space/datafile 2024-04-25T08:27:02.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:02.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj340 td/osd-recovery-space/datafile 2024-04-25T08:27:02.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:02.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj341 td/osd-recovery-space/datafile 2024-04-25T08:27:02.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:02.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj342 td/osd-recovery-space/datafile 2024-04-25T08:27:02.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:02.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj343 td/osd-recovery-space/datafile 2024-04-25T08:27:02.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:02.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj344 td/osd-recovery-space/datafile 2024-04-25T08:27:02.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:02.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj345 td/osd-recovery-space/datafile 2024-04-25T08:27:02.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:02.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj346 td/osd-recovery-space/datafile 2024-04-25T08:27:03.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj347 td/osd-recovery-space/datafile 2024-04-25T08:27:03.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj348 td/osd-recovery-space/datafile 2024-04-25T08:27:03.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj349 td/osd-recovery-space/datafile 2024-04-25T08:27:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj350 td/osd-recovery-space/datafile 2024-04-25T08:27:03.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj351 td/osd-recovery-space/datafile 2024-04-25T08:27:03.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj352 td/osd-recovery-space/datafile 2024-04-25T08:27:03.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj353 td/osd-recovery-space/datafile 2024-04-25T08:27:03.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj354 td/osd-recovery-space/datafile 2024-04-25T08:27:03.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj355 td/osd-recovery-space/datafile 2024-04-25T08:27:03.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj356 td/osd-recovery-space/datafile 2024-04-25T08:27:03.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj357 td/osd-recovery-space/datafile 2024-04-25T08:27:03.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:03.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj358 td/osd-recovery-space/datafile 2024-04-25T08:27:04.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj359 td/osd-recovery-space/datafile 2024-04-25T08:27:04.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj360 td/osd-recovery-space/datafile 2024-04-25T08:27:04.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj361 td/osd-recovery-space/datafile 2024-04-25T08:27:04.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj362 td/osd-recovery-space/datafile 2024-04-25T08:27:04.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj363 td/osd-recovery-space/datafile 2024-04-25T08:27:04.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj364 td/osd-recovery-space/datafile 2024-04-25T08:27:04.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj365 td/osd-recovery-space/datafile 2024-04-25T08:27:04.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj366 td/osd-recovery-space/datafile 2024-04-25T08:27:04.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj367 td/osd-recovery-space/datafile 2024-04-25T08:27:04.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj368 td/osd-recovery-space/datafile 2024-04-25T08:27:04.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj369 td/osd-recovery-space/datafile 2024-04-25T08:27:04.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj370 td/osd-recovery-space/datafile 2024-04-25T08:27:04.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj371 td/osd-recovery-space/datafile 2024-04-25T08:27:04.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:04.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj372 td/osd-recovery-space/datafile 2024-04-25T08:27:05.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj373 td/osd-recovery-space/datafile 2024-04-25T08:27:05.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj374 td/osd-recovery-space/datafile 2024-04-25T08:27:05.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj375 td/osd-recovery-space/datafile 2024-04-25T08:27:05.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj376 td/osd-recovery-space/datafile 2024-04-25T08:27:05.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj377 td/osd-recovery-space/datafile 2024-04-25T08:27:05.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj378 td/osd-recovery-space/datafile 2024-04-25T08:27:05.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj379 td/osd-recovery-space/datafile 2024-04-25T08:27:05.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj380 td/osd-recovery-space/datafile 2024-04-25T08:27:05.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj381 td/osd-recovery-space/datafile 2024-04-25T08:27:05.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj382 td/osd-recovery-space/datafile 2024-04-25T08:27:05.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj383 td/osd-recovery-space/datafile 2024-04-25T08:27:05.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj384 td/osd-recovery-space/datafile 2024-04-25T08:27:05.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:05.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj385 td/osd-recovery-space/datafile 2024-04-25T08:27:06.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj386 td/osd-recovery-space/datafile 2024-04-25T08:27:06.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj387 td/osd-recovery-space/datafile 2024-04-25T08:27:06.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj388 td/osd-recovery-space/datafile 2024-04-25T08:27:06.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj389 td/osd-recovery-space/datafile 2024-04-25T08:27:06.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj390 td/osd-recovery-space/datafile 2024-04-25T08:27:06.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj391 td/osd-recovery-space/datafile 2024-04-25T08:27:06.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj392 td/osd-recovery-space/datafile 2024-04-25T08:27:06.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj393 td/osd-recovery-space/datafile 2024-04-25T08:27:06.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj394 td/osd-recovery-space/datafile 2024-04-25T08:27:06.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj395 td/osd-recovery-space/datafile 2024-04-25T08:27:06.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj396 td/osd-recovery-space/datafile 2024-04-25T08:27:06.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:06.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj397 td/osd-recovery-space/datafile 2024-04-25T08:27:07.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj398 td/osd-recovery-space/datafile 2024-04-25T08:27:07.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj399 td/osd-recovery-space/datafile 2024-04-25T08:27:07.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj400 td/osd-recovery-space/datafile 2024-04-25T08:27:07.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj401 td/osd-recovery-space/datafile 2024-04-25T08:27:07.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj402 td/osd-recovery-space/datafile 2024-04-25T08:27:07.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj403 td/osd-recovery-space/datafile 2024-04-25T08:27:07.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj404 td/osd-recovery-space/datafile 2024-04-25T08:27:07.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj405 td/osd-recovery-space/datafile 2024-04-25T08:27:07.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj406 td/osd-recovery-space/datafile 2024-04-25T08:27:07.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj407 td/osd-recovery-space/datafile 2024-04-25T08:27:07.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj408 td/osd-recovery-space/datafile 2024-04-25T08:27:07.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj409 td/osd-recovery-space/datafile 2024-04-25T08:27:07.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:07.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj410 td/osd-recovery-space/datafile 2024-04-25T08:27:07.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj411 td/osd-recovery-space/datafile 2024-04-25T08:27:08.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj412 td/osd-recovery-space/datafile 2024-04-25T08:27:08.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj413 td/osd-recovery-space/datafile 2024-04-25T08:27:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj414 td/osd-recovery-space/datafile 2024-04-25T08:27:08.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj415 td/osd-recovery-space/datafile 2024-04-25T08:27:08.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj416 td/osd-recovery-space/datafile 2024-04-25T08:27:08.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj417 td/osd-recovery-space/datafile 2024-04-25T08:27:08.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj418 td/osd-recovery-space/datafile 2024-04-25T08:27:08.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj419 td/osd-recovery-space/datafile 2024-04-25T08:27:08.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj420 td/osd-recovery-space/datafile 2024-04-25T08:27:08.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj421 td/osd-recovery-space/datafile 2024-04-25T08:27:08.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:08.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj422 td/osd-recovery-space/datafile 2024-04-25T08:27:09.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:09.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj423 td/osd-recovery-space/datafile 2024-04-25T08:27:09.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:09.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj424 td/osd-recovery-space/datafile 2024-04-25T08:27:09.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:09.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj425 td/osd-recovery-space/datafile 2024-04-25T08:27:09.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:09.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj426 td/osd-recovery-space/datafile 2024-04-25T08:27:09.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:09.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj427 td/osd-recovery-space/datafile 2024-04-25T08:27:09.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:09.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj428 td/osd-recovery-space/datafile 2024-04-25T08:27:09.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:09.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj429 td/osd-recovery-space/datafile 2024-04-25T08:27:09.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:09.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj430 td/osd-recovery-space/datafile 2024-04-25T08:27:09.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:09.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj431 td/osd-recovery-space/datafile 2024-04-25T08:27:09.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:09.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj432 td/osd-recovery-space/datafile 2024-04-25T08:27:09.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:09.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj433 td/osd-recovery-space/datafile 2024-04-25T08:27:10.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj434 td/osd-recovery-space/datafile 2024-04-25T08:27:10.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj435 td/osd-recovery-space/datafile 2024-04-25T08:27:10.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj436 td/osd-recovery-space/datafile 2024-04-25T08:27:10.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj437 td/osd-recovery-space/datafile 2024-04-25T08:27:10.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj438 td/osd-recovery-space/datafile 2024-04-25T08:27:10.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj439 td/osd-recovery-space/datafile 2024-04-25T08:27:10.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj440 td/osd-recovery-space/datafile 2024-04-25T08:27:10.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj441 td/osd-recovery-space/datafile 2024-04-25T08:27:10.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj442 td/osd-recovery-space/datafile 2024-04-25T08:27:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj443 td/osd-recovery-space/datafile 2024-04-25T08:27:10.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj444 td/osd-recovery-space/datafile 2024-04-25T08:27:10.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj445 td/osd-recovery-space/datafile 2024-04-25T08:27:10.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:10.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj446 td/osd-recovery-space/datafile 2024-04-25T08:27:11.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj447 td/osd-recovery-space/datafile 2024-04-25T08:27:11.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj448 td/osd-recovery-space/datafile 2024-04-25T08:27:11.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj449 td/osd-recovery-space/datafile 2024-04-25T08:27:11.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj450 td/osd-recovery-space/datafile 2024-04-25T08:27:11.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj451 td/osd-recovery-space/datafile 2024-04-25T08:27:11.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj452 td/osd-recovery-space/datafile 2024-04-25T08:27:11.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj453 td/osd-recovery-space/datafile 2024-04-25T08:27:11.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj454 td/osd-recovery-space/datafile 2024-04-25T08:27:11.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj455 td/osd-recovery-space/datafile 2024-04-25T08:27:11.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj456 td/osd-recovery-space/datafile 2024-04-25T08:27:11.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj457 td/osd-recovery-space/datafile 2024-04-25T08:27:11.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj458 td/osd-recovery-space/datafile 2024-04-25T08:27:11.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:11.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj459 td/osd-recovery-space/datafile 2024-04-25T08:27:12.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:12.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj460 td/osd-recovery-space/datafile 2024-04-25T08:27:12.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:12.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj461 td/osd-recovery-space/datafile 2024-04-25T08:27:12.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:12.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj462 td/osd-recovery-space/datafile 2024-04-25T08:27:12.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:12.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj463 td/osd-recovery-space/datafile 2024-04-25T08:27:12.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:12.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj464 td/osd-recovery-space/datafile 2024-04-25T08:27:12.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:12.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj465 td/osd-recovery-space/datafile 2024-04-25T08:27:12.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:12.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj466 td/osd-recovery-space/datafile 2024-04-25T08:27:12.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:12.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj467 td/osd-recovery-space/datafile 2024-04-25T08:27:12.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:12.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj468 td/osd-recovery-space/datafile 2024-04-25T08:27:12.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:12.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj469 td/osd-recovery-space/datafile 2024-04-25T08:27:12.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:12.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj470 td/osd-recovery-space/datafile 2024-04-25T08:27:13.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:13.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj471 td/osd-recovery-space/datafile 2024-04-25T08:27:13.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:13.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj472 td/osd-recovery-space/datafile 2024-04-25T08:27:13.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:13.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj473 td/osd-recovery-space/datafile 2024-04-25T08:27:13.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:13.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj474 td/osd-recovery-space/datafile 2024-04-25T08:27:13.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:13.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj475 td/osd-recovery-space/datafile 2024-04-25T08:27:13.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:13.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj476 td/osd-recovery-space/datafile 2024-04-25T08:27:13.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:13.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj477 td/osd-recovery-space/datafile 2024-04-25T08:27:13.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:13.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj478 td/osd-recovery-space/datafile 2024-04-25T08:27:13.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:13.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj479 td/osd-recovery-space/datafile 2024-04-25T08:27:13.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:13.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj480 td/osd-recovery-space/datafile 2024-04-25T08:27:13.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:13.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj481 td/osd-recovery-space/datafile 2024-04-25T08:27:14.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj482 td/osd-recovery-space/datafile 2024-04-25T08:27:14.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj483 td/osd-recovery-space/datafile 2024-04-25T08:27:14.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj484 td/osd-recovery-space/datafile 2024-04-25T08:27:14.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj485 td/osd-recovery-space/datafile 2024-04-25T08:27:14.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj486 td/osd-recovery-space/datafile 2024-04-25T08:27:14.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj487 td/osd-recovery-space/datafile 2024-04-25T08:27:14.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj488 td/osd-recovery-space/datafile 2024-04-25T08:27:14.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj489 td/osd-recovery-space/datafile 2024-04-25T08:27:14.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj490 td/osd-recovery-space/datafile 2024-04-25T08:27:14.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj491 td/osd-recovery-space/datafile 2024-04-25T08:27:14.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj492 td/osd-recovery-space/datafile 2024-04-25T08:27:14.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:14.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj493 td/osd-recovery-space/datafile 2024-04-25T08:27:15.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:15.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj494 td/osd-recovery-space/datafile 2024-04-25T08:27:15.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:15.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj495 td/osd-recovery-space/datafile 2024-04-25T08:27:15.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:15.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj496 td/osd-recovery-space/datafile 2024-04-25T08:27:15.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:15.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj497 td/osd-recovery-space/datafile 2024-04-25T08:27:15.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:15.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj498 td/osd-recovery-space/datafile 2024-04-25T08:27:15.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:15.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj499 td/osd-recovery-space/datafile 2024-04-25T08:27:15.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:15.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj500 td/osd-recovery-space/datafile 2024-04-25T08:27:15.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:15.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj501 td/osd-recovery-space/datafile 2024-04-25T08:27:15.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:15.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj502 td/osd-recovery-space/datafile 2024-04-25T08:27:15.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:15.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj503 td/osd-recovery-space/datafile 2024-04-25T08:27:15.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:15.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj504 td/osd-recovery-space/datafile 2024-04-25T08:27:16.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj505 td/osd-recovery-space/datafile 2024-04-25T08:27:16.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj506 td/osd-recovery-space/datafile 2024-04-25T08:27:16.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj507 td/osd-recovery-space/datafile 2024-04-25T08:27:16.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj508 td/osd-recovery-space/datafile 2024-04-25T08:27:16.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj509 td/osd-recovery-space/datafile 2024-04-25T08:27:16.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj510 td/osd-recovery-space/datafile 2024-04-25T08:27:16.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj511 td/osd-recovery-space/datafile 2024-04-25T08:27:16.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj512 td/osd-recovery-space/datafile 2024-04-25T08:27:16.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj513 td/osd-recovery-space/datafile 2024-04-25T08:27:16.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj514 td/osd-recovery-space/datafile 2024-04-25T08:27:16.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj515 td/osd-recovery-space/datafile 2024-04-25T08:27:16.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:16.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj516 td/osd-recovery-space/datafile 2024-04-25T08:27:17.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj517 td/osd-recovery-space/datafile 2024-04-25T08:27:17.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj518 td/osd-recovery-space/datafile 2024-04-25T08:27:17.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj519 td/osd-recovery-space/datafile 2024-04-25T08:27:17.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj520 td/osd-recovery-space/datafile 2024-04-25T08:27:17.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj521 td/osd-recovery-space/datafile 2024-04-25T08:27:17.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj522 td/osd-recovery-space/datafile 2024-04-25T08:27:17.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj523 td/osd-recovery-space/datafile 2024-04-25T08:27:17.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj524 td/osd-recovery-space/datafile 2024-04-25T08:27:17.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj525 td/osd-recovery-space/datafile 2024-04-25T08:27:17.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj526 td/osd-recovery-space/datafile 2024-04-25T08:27:17.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj527 td/osd-recovery-space/datafile 2024-04-25T08:27:17.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:17.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj528 td/osd-recovery-space/datafile 2024-04-25T08:27:18.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:18.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj529 td/osd-recovery-space/datafile 2024-04-25T08:27:18.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:18.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj530 td/osd-recovery-space/datafile 2024-04-25T08:27:18.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:18.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj531 td/osd-recovery-space/datafile 2024-04-25T08:27:18.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:18.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj532 td/osd-recovery-space/datafile 2024-04-25T08:27:18.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:18.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj533 td/osd-recovery-space/datafile 2024-04-25T08:27:18.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:18.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj534 td/osd-recovery-space/datafile 2024-04-25T08:27:18.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:18.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj535 td/osd-recovery-space/datafile 2024-04-25T08:27:18.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:18.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj536 td/osd-recovery-space/datafile 2024-04-25T08:27:18.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:18.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj537 td/osd-recovery-space/datafile 2024-04-25T08:27:18.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:18.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj538 td/osd-recovery-space/datafile 2024-04-25T08:27:19.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj539 td/osd-recovery-space/datafile 2024-04-25T08:27:19.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj540 td/osd-recovery-space/datafile 2024-04-25T08:27:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj541 td/osd-recovery-space/datafile 2024-04-25T08:27:19.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj542 td/osd-recovery-space/datafile 2024-04-25T08:27:19.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj543 td/osd-recovery-space/datafile 2024-04-25T08:27:19.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj544 td/osd-recovery-space/datafile 2024-04-25T08:27:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj545 td/osd-recovery-space/datafile 2024-04-25T08:27:19.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj546 td/osd-recovery-space/datafile 2024-04-25T08:27:19.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj547 td/osd-recovery-space/datafile 2024-04-25T08:27:19.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj548 td/osd-recovery-space/datafile 2024-04-25T08:27:19.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj549 td/osd-recovery-space/datafile 2024-04-25T08:27:19.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj550 td/osd-recovery-space/datafile 2024-04-25T08:27:19.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:19.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj551 td/osd-recovery-space/datafile 2024-04-25T08:27:20.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj552 td/osd-recovery-space/datafile 2024-04-25T08:27:20.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj553 td/osd-recovery-space/datafile 2024-04-25T08:27:20.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj554 td/osd-recovery-space/datafile 2024-04-25T08:27:20.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj555 td/osd-recovery-space/datafile 2024-04-25T08:27:20.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj556 td/osd-recovery-space/datafile 2024-04-25T08:27:20.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj557 td/osd-recovery-space/datafile 2024-04-25T08:27:20.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj558 td/osd-recovery-space/datafile 2024-04-25T08:27:20.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj559 td/osd-recovery-space/datafile 2024-04-25T08:27:20.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj560 td/osd-recovery-space/datafile 2024-04-25T08:27:20.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj561 td/osd-recovery-space/datafile 2024-04-25T08:27:20.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj562 td/osd-recovery-space/datafile 2024-04-25T08:27:20.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:20.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj563 td/osd-recovery-space/datafile 2024-04-25T08:27:21.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:21.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj564 td/osd-recovery-space/datafile 2024-04-25T08:27:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:21.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj565 td/osd-recovery-space/datafile 2024-04-25T08:27:21.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:21.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj566 td/osd-recovery-space/datafile 2024-04-25T08:27:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj567 td/osd-recovery-space/datafile 2024-04-25T08:27:21.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:21.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj568 td/osd-recovery-space/datafile 2024-04-25T08:27:21.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:21.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj569 td/osd-recovery-space/datafile 2024-04-25T08:27:21.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:21.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj570 td/osd-recovery-space/datafile 2024-04-25T08:27:21.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:21.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj571 td/osd-recovery-space/datafile 2024-04-25T08:27:21.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:21.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj572 td/osd-recovery-space/datafile 2024-04-25T08:27:21.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:21.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj573 td/osd-recovery-space/datafile 2024-04-25T08:27:22.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj574 td/osd-recovery-space/datafile 2024-04-25T08:27:22.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj575 td/osd-recovery-space/datafile 2024-04-25T08:27:22.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj576 td/osd-recovery-space/datafile 2024-04-25T08:27:22.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj577 td/osd-recovery-space/datafile 2024-04-25T08:27:22.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj578 td/osd-recovery-space/datafile 2024-04-25T08:27:22.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj579 td/osd-recovery-space/datafile 2024-04-25T08:27:22.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj580 td/osd-recovery-space/datafile 2024-04-25T08:27:22.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj581 td/osd-recovery-space/datafile 2024-04-25T08:27:22.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj582 td/osd-recovery-space/datafile 2024-04-25T08:27:22.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj583 td/osd-recovery-space/datafile 2024-04-25T08:27:22.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj584 td/osd-recovery-space/datafile 2024-04-25T08:27:22.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj585 td/osd-recovery-space/datafile 2024-04-25T08:27:22.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:22.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj586 td/osd-recovery-space/datafile 2024-04-25T08:27:23.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj587 td/osd-recovery-space/datafile 2024-04-25T08:27:23.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj588 td/osd-recovery-space/datafile 2024-04-25T08:27:23.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj589 td/osd-recovery-space/datafile 2024-04-25T08:27:23.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj590 td/osd-recovery-space/datafile 2024-04-25T08:27:23.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj591 td/osd-recovery-space/datafile 2024-04-25T08:27:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj592 td/osd-recovery-space/datafile 2024-04-25T08:27:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj593 td/osd-recovery-space/datafile 2024-04-25T08:27:23.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj594 td/osd-recovery-space/datafile 2024-04-25T08:27:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj595 td/osd-recovery-space/datafile 2024-04-25T08:27:23.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj596 td/osd-recovery-space/datafile 2024-04-25T08:27:23.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj597 td/osd-recovery-space/datafile 2024-04-25T08:27:23.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:23.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj598 td/osd-recovery-space/datafile 2024-04-25T08:27:24.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:24.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj599 td/osd-recovery-space/datafile 2024-04-25T08:27:24.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:27:24.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj600 td/osd-recovery-space/datafile 2024-04-25T08:27:24.262 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: expr 2 - 1 2024-04-25T08:27:24.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: seq 0 1 2024-04-25T08:27:24.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:27:24.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.0 injectargs '--fake_statfs_for_testing 3686400' 2024-04-25T08:27:24.410 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T08:27:24.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:27:24.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.1 injectargs '--fake_statfs_for_testing 3686400' 2024-04-25T08:27:24.575 INFO:tasks.workunit.client.0.smithi102.stdout:{} 2024-04-25T08:27:24.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: sleep 5 2024-04-25T08:27:29.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:126: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:27:29.994 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:27:29.994 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 600 0 0 0 0 3072000 0 0 600 0 600 active+clean 2024-04-25T08:26:20.143355+0000 23'600 24:620 [1] 1 [1] 1 0'0 2024-04-25T08:26:17.069969+0000 0'0 2024-04-25T08:26:17.069969+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:31:09.610600+0000 0 0 2024-04-25T08:27:29.994 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:27:29.994 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:27:29.994 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:27:30.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:27:30.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-04-25T08:27:30.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:130: TEST_recovery_test_simple: ceph osd pool set test1 size 2 2024-04-25T08:27:30.748 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-04-25T08:27:30.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:134: TEST_recovery_test_simple: wait_for_recovery_toofull 30 2024-04-25T08:27:30.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:77: wait_for_recovery_toofull: local timeout=30 2024-04-25T08:27:30.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:78: wait_for_recovery_toofull: wait_for_state recovery_toofull 30 2024-04-25T08:27:30.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:52: wait_for_state: local state=recovery_toofull 2024-04-25T08:27:30.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: local cur_in_state 2024-04-25T08:27:30.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: get_timeout_delays 30 5 2024-04-25T08:27:30.777 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:27:30.777 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:27:30.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:27:30.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:27:30.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:27:30.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: delays=('5' '10' '15') 2024-04-25T08:27:30.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: local -a delays 2024-04-25T08:27:30.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:55: wait_for_state: local -i loop=0 2024-04-25T08:27:30.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: flush_pg_stats 2024-04-25T08:27:30.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:27:30.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:27:31.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:27:31.286 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-04-25T08:27:31.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:27:31.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:27:31.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:27:31.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836512 2024-04-25T08:27:31.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836512 2024-04-25T08:27:31.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512' 2024-04-25T08:27:31.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:27:31.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:27:31.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-04-25T08:27:31.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-04-25T08:27:31.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672985' 2024-04-25T08:27:31.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:27:31.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836512 2024-04-25T08:27:31.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:27:31.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:27:31.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836512 2024-04-25T08:27:31.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:27:31.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836512 2024-04-25T08:27:31.730 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836512 2024-04-25T08:27:31.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-04-25T08:27:31.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:27:32.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836512 2024-04-25T08:27:32.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:27:33.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:27:33.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:27:33.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836512 2024-04-25T08:27:33.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:27:34.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-25T08:27:34.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:27:35.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836512 -lt 21474836512 2024-04-25T08:27:35.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:27:35.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-04-25T08:27:35.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:27:35.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:27:35.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-04-25T08:27:35.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:27:35.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-04-25T08:27:35.007 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672985 2024-04-25T08:27:35.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-25T08:27:35.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:27:35.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-04-25T08:27:35.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: get_num_pgs 2024-04-25T08:27:35.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:35.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:36.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: test 1 == 0 2024-04-25T08:27:36.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: true 2024-04-25T08:27:36.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: get_num_in_state recovery_toofull 2024-04-25T08:27:36.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local state=recovery_toofull 2024-04-25T08:27:36.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: local expression 2024-04-25T08:27:36.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2024-04-25T08:27:36.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: ceph --format json pg dump pgs 2024-04-25T08:27:36.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:47: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2024-04-25T08:27:36.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: cur_in_state=1 2024-04-25T08:27:36.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: test 1 -gt 0 2024-04-25T08:27:36.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: break 2024-04-25T08:27:36.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:72: wait_for_state: return 0 2024-04-25T08:27:36.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: ERRORS=0 2024-04-25T08:27:36.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:27:36.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: grep +recovery_toofull 2024-04-25T08:27:36.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: wc -l 2024-04-25T08:27:36.874 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:27:36.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: '[' 1 '!=' 1 ']' 2024-04-25T08:27:36.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:143: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:27:37.261 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:27:37.261 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 600 0 600 0 0 3072000 0 0 600 0 600 active+recovery_toofull+undersized+degraded+remapped 2024-04-25T08:27:31.540198+0000 23'600 27:628 [1,0] 1 [1] 1 0'0 2024-04-25T08:26:17.069969+0000 0'0 2024-04-25T08:26:17.069969+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:27:37.261 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:27:37.262 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:27:37.262 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T08:27:37.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:144: TEST_recovery_test_simple: ceph status 2024-04-25T08:27:37.815 INFO:tasks.workunit.client.0.smithi102.stdout: cluster: 2024-04-25T08:27:37.815 INFO:tasks.workunit.client.0.smithi102.stdout: id: c6ba824e-b840-43fe-93b6-97665ee7e006 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: health: HEALTH_ERR 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: 2 full osd(s) 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: Degraded data redundancy: 600/1200 objects degraded (50.000%), 1 pg degraded 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: Full OSDs blocking recovery: 1 pg recovery_toofull 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: 1 pool(s) do not have an application enabled 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: 1 pool(s) full 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: OSD count 2 < osd_pool_default_size 3 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: services: 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: mon: 1 daemons, quorum a (age 2m) 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: mgr: x(active, since 2m) 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: osd: 2 osds: 2 up (since 84s), 2 in (since 2m); 1 remapped pgs 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: data: 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: pools: 1 pools, 1 pgs 2024-04-25T08:27:37.816 INFO:tasks.workunit.client.0.smithi102.stdout: objects: 600 objects, 2.9 MiB 2024-04-25T08:27:37.817 INFO:tasks.workunit.client.0.smithi102.stdout: usage: 5.9 MiB used, 1.2 MiB / 7.0 MiB avail 2024-04-25T08:27:37.817 INFO:tasks.workunit.client.0.smithi102.stdout: pgs: 600/1200 objects degraded (50.000%) 2024-04-25T08:27:37.817 INFO:tasks.workunit.client.0.smithi102.stdout: 1 active+recovery_toofull+undersized+degraded+remapped 2024-04-25T08:27:37.817 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T08:27:37.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: ceph status --format=json-pretty 2024-04-25T08:27:38.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.severity td/osd-recovery-space/stat.json 2024-04-25T08:27:38.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: eval 'SEV="HEALTH_ERR"' 2024-04-25T08:27:38.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: SEV=HEALTH_ERR 2024-04-25T08:27:38.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:148: TEST_recovery_test_simple: '[' HEALTH_ERR '!=' HEALTH_ERR ']' 2024-04-25T08:27:38.433 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.summary.message td/osd-recovery-space/stat.json 2024-04-25T08:27:38.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: eval 'MSG="Full' OSDs blocking recovery: 1 pg 'recovery_toofull"' 2024-04-25T08:27:38.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: MSG='Full OSDs blocking recovery: 1 pg recovery_toofull' 2024-04-25T08:27:38.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:153: TEST_recovery_test_simple: '[' 'Full OSDs blocking recovery: 1 pg recovery_toofull' '!=' 'Full OSDs blocking recovery: 1 pg recovery_toofull' ']' 2024-04-25T08:27:38.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:157: TEST_recovery_test_simple: rm -f td/osd-recovery-space/stat.json 2024-04-25T08:27:38.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:159: TEST_recovery_test_simple: '[' 0 '!=' 0 ']' 2024-04-25T08:27:38.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:164: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:27:38.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:164: TEST_recovery_test_simple: for i in $(seq 1 $pools) 2024-04-25T08:27:38.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:166: TEST_recovery_test_simple: delete_pool test1 2024-04-25T08:27:38.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T08:27:38.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T08:27:38.974 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test1' does not exist 2024-04-25T08:27:38.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:168: TEST_recovery_test_simple: kill_daemons td/osd-recovery-space 2024-04-25T08:27:38.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:27:38.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:27:38.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:27:38.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:27:38.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:27:44.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:27:44.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:37: run: teardown td/osd-recovery-space 2024-04-25T08:27:44.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:27:44.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:27:44.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:27:44.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:27:44.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:27:44.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:27:44.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:27:44.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:27:44.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:27:44.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:27:44.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:27:44.317 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:27:44.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:27:44.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:27:44.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:27:44.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:27:44.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:27:44.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:27:44.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:27:44.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:27:44.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:27:44.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:27:44.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:27:44.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:27:44.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.260802 2024-04-25T08:27:44.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.260802 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-space 0 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:27:44.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:27:44.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:27:44.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:27:44.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:27:44.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:27:44.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:27:44.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:27:44.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:27:44.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:27:44.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:27:44.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:27:44.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:27:44.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:27:44.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:27:44.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:27:44.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:27:44.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:27:44.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.260802 2024-04-25T08:27:44.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.260802 2024-04-25T08:27:44.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:27:44.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:27:44.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:27:44.375 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:27:44.375 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:27:44.387 INFO:tasks.workunit:Running workunit osd/osd-recovery-stats.sh... 2024-04-25T08:27:44.387 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-recovery-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh 2024-04-25T08:27:44.438 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-stats 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:21: run: local dir=td/osd-recovery-stats 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:22: run: shift 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T08:27:44.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: CEPH_MON=127.0.0.1:7115 2024-04-25T08:27:44.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:26: run: export CEPH_ARGS 2024-04-25T08:27:44.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: uuidgen 2024-04-25T08:27:44.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: CEPH_ARGS+='--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none ' 2024-04-25T08:27:44.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T08:27:44.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:30: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:27:44.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: export margin=10 2024-04-25T08:27:44.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: margin=10 2024-04-25T08:27:44.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: export objects=200 2024-04-25T08:27:44.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: objects=200 2024-04-25T08:27:44.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: export poolname=test 2024-04-25T08:27:44.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: poolname=test 2024-04-25T08:27:44.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:27:44.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: set 2024-04-25T08:27:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: local 'funcs=TEST_recovery_erasure_out1 2024-04-25T08:27:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_recovery_erasure_remapped 2024-04-25T08:27:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_recovery_multi 2024-04-25T08:27:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_recovery_replicated_out1 2024-04-25T08:27:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_recovery_sizedown 2024-04-25T08:27:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_recovery_sizeup 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_recovery_undersized' 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:27:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:27:44.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:27:44.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:27:44.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:27:44.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:27:44.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:27:44.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:27:44.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:27:44.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:27:44.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:27:44.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:27:44.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:27:44.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:27:44.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:27:44.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:27:44.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:27:44.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:27:44.460 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:27:44.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:27:44.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:27:44.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:27:44.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:27:44.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:27:44.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:27:44.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:27:44.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.277516 2024-04-25T08:27:44.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:27:44.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:27:44.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:27:44.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:27:44.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:27:44.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_out1 td/osd-recovery-stats 2024-04-25T08:27:44.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:173: TEST_recovery_erasure_out1: local dir=td/osd-recovery-stats 2024-04-25T08:27:44.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:175: TEST_recovery_erasure_out1: do_recovery_out1 td/osd-recovery-stats erasure 2024-04-25T08:27:44.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-04-25T08:27:44.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-04-25T08:27:44.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=erasure 2024-04-25T08:27:44.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-04-25T08:27:44.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:27:44.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:27:44.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:27:44.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:27:44.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:27:44.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:27:44.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:27:44.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:27:44.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:27:44.965 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:27:44.965 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:27:44.965 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:27:44.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:27:44.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:27:45.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:27:45.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:27:45.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:27:45.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:27:45.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:27:45.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:27:45.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:27:45.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:27:45.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:27:45.187 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:27:45.187 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:27:45.187 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:27:45.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:27:45.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:27:45.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get fsid 2024-04-25T08:27:45.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:27:45.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:27:45.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:27:45.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:27:45.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:27:45.305 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:27:45.305 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:27:45.305 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:27:45.306 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:27:45.306 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:27:45.306 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:27:45.306 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:27:45.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:27:45.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get mon_host 2024-04-25T08:27:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-04-25T08:27:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:27:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:27:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:27:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:27:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:27:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:27:46.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:27:46.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:27:46.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:27:46.330 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:27:46.330 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:27:46.330 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:27:46.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:27:46.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:27:46.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:27:46.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-04-25T08:27:46.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:27:46.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:27:46.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:27:46.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:27:46.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:27:46.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:27:46.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef448256-a7e2-433e-8f14-3169ffd4f0f9 2024-04-25T08:27:46.370 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 ef448256-a7e2-433e-8f14-3169ffd4f0f9 2024-04-25T08:27:46.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ef448256-a7e2-433e-8f14-3169ffd4f0f9' 2024-04-25T08:27:46.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:27:46.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQACFCpmA5dAFxAAN/VDDqgb2fMO3Brh8YvAug== 2024-04-25T08:27:46.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQACFCpmA5dAFxAAN/VDDqgb2fMO3Brh8YvAug=="}' 2024-04-25T08:27:46.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef448256-a7e2-433e-8f14-3169ffd4f0f9 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:27:46.704 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:27:46.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:27:46.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQACFCpmA5dAFxAAN/VDDqgb2fMO3Brh8YvAug== --osd-uuid ef448256-a7e2-433e-8f14-3169ffd4f0f9 2024-04-25T08:27:46.749 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:46.749+0000 7fa734b757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:46.781 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:46.781+0000 7fa734b757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:46.782 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:46.781+0000 7fa734b757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:46.783 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:46.781+0000 7fa734b757c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:27:46.783 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:46.781+0000 7fa734b757c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:27:46.783 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:46.781+0000 7fa734b757c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:27:46.821 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:46.821+0000 7fa734b757c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:27:53.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:27:53.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:27:53.315 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:27:53.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:27:53.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:27:53.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:27:53.871 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:27:53.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:27:53.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:27:53.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:27:53.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:27:53.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:53.900+0000 7f7c36b697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:53.946 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:53.944+0000 7f7c36b697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:53.948 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:53.948+0000 7f7c36b697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:27:54.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:27:54.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:27:54.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:27:54.311 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:27:54.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:27:54.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:27:54.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:54.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:27:54.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:54.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:27:54.517 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:54.516+0000 7f7c36b697c0 -1 Falling back to public interface 2024-04-25T08:27:54.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:55.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:55.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:55.736 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:27:55.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:27:55.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:55.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:27:56.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:57.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:57.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:57.145 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:27:57.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:27:57.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:57.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:27:57.167 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:27:57.164+0000 7f7c36b697c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:27:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:27:58.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:27:58.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:27:58.727 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:27:58.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:27:58.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:27:58.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:27:59.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:00.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:00.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:00.181 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:28:00.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:28:00.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:00.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:00.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:01.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:01.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:01.595 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:28:01.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:28:01.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:01.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:02.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:03.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:03.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:03.183 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:28:03.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:28:03.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:03.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:03.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:04.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:04.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:04.629 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:28:04.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:28:04.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:04.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:05.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:06.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:06.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:06.152 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:28:06.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:28:06.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:06.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:06.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:07.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:07.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:07.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:28:07.565 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:28:07.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:07.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:08.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:09.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:09.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:09.058 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:28:09.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:28:09.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:09.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:09.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:10.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:10.546 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:28:10.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:10.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:28:10.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:10.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:11.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:12.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:12.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:12.048 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:28:12.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:28:12.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:12.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:13.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:14.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:14.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:14.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:28:14.006 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:28:14.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:14.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:14.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:15.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:15.860 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:28:15.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:28:15.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:15.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:16.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:17.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:17.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:17.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:28:17.932 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:28:17.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:17.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:18.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:19.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:19.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:19.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:28:19.790 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:28:19.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:19.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:20.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:21.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:21.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:21.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:28:21.361 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:28:21.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:21.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:22.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:23.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:23.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:23.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:28:23.067 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:28:23.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:23.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:28:23.493 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1173416624,v1:127.0.0.1:6803/1173416624] [v2:127.0.0.1:6804/1173416624,v1:127.0.0.1:6805/1173416624] exists,up ef448256-a7e2-433e-8f14-3169ffd4f0f9 2024-04-25T08:28:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:28:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:28:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:28:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-04-25T08:28:23.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:28:23.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:28:23.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:28:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:28:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:28:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:28:23.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:28:23.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1b5fea43-6763-4727-867b-57ce924fef77 2024-04-25T08:28:23.500 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 1b5fea43-6763-4727-867b-57ce924fef77 2024-04-25T08:28:23.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1b5fea43-6763-4727-867b-57ce924fef77' 2024-04-25T08:28:23.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:28:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnFCpmbPEPHxAAAY/Nvt1EdamSqQzdN7omhg== 2024-04-25T08:28:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnFCpmbPEPHxAAAY/Nvt1EdamSqQzdN7omhg=="}' 2024-04-25T08:28:23.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1b5fea43-6763-4727-867b-57ce924fef77 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:28:24.027 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:28:24.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:28:24.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAnFCpmbPEPHxAAAY/Nvt1EdamSqQzdN7omhg== --osd-uuid 1b5fea43-6763-4727-867b-57ce924fef77 2024-04-25T08:28:24.079 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:24.076+0000 7f46acb6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:24.104 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:24.104+0000 7f46acb6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:24.105 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:24.104+0000 7f46acb6d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:24.106 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:24.104+0000 7f46acb6d7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:28:24.106 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:24.104+0000 7f46acb6d7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:28:24.106 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:24.104+0000 7f46acb6d7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:28:24.151 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:24.152+0000 7f46acb6d7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:28:31.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:28:31.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:28:31.058 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T08:28:31.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:28:31.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:28:31.650 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:28:31.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:28:31.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:28:31.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:28:31.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:28:31.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:28:31.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:31.680+0000 7f5dde9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:31.712 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:31.712+0000 7f5dde9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:31.713 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:31.712+0000 7f5dde9437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:28:32.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:28:32.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:28:32.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:28:32.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:28:32.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:28:32.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:32.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:28:32.090 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:28:32.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:32.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:32.261 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:32.260+0000 7f5dde9437c0 -1 Falling back to public interface 2024-04-25T08:28:32.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:33.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:33.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:33.474 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:28:33.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:28:33.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:33.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:33.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:34.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:34.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:34.880 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:28:34.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:28:34.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:34.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:34.975 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:28:34.972+0000 7f5dde9437c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:28:35.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:36.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:36.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:36.389 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:28:36.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:28:36.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:36.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:36.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:37.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:37.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:37.835 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:28:37.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:28:37.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:37.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:38.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:39.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:39.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:39.524 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:28:39.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:28:39.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:39.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:40.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:41.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:41.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:41.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:28:41.134 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:28:41.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:41.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:41.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:42.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:42.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:42.772 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:28:42.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:28:42.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:42.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:43.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:44.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:44.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:44.504 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:28:44.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:28:44.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:44.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:45.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:46.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:46.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:46.215 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:28:46.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:28:46.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:46.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:46.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:47.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:47.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:47.646 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:28:47.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:28:47.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:47.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:48.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:49.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:49.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:49.156 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:28:49.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:28:49.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:49.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:49.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:50.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:50.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:50.762 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:28:50.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:28:50.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:50.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:51.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:52.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:52.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:52.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:28:52.858 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:28:52.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:52.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:54.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:55.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:55.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:55.400 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:28:55.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:28:55.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:55.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:56.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:57.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:57.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:57.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:28:57.263 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:28:57.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:57.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:28:58.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:28:59.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:28:59.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:28:59.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:28:59.319 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:28:59.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:28:59.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:00.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:01.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:01.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:01.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:29:01.256 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:29:01.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:01.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:02.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:03.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:03.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:03.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:29:03.106 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:29:03.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:03.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:29:03.540 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/158390743,v1:127.0.0.1:6811/158390743] [v2:127.0.0.1:6812/158390743,v1:127.0.0.1:6813/158390743] exists,up 1b5fea43-6763-4727-867b-57ce924fef77 2024-04-25T08:29:03.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:29:03.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:29:03.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:29:03.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:29:03.544 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:29:03.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:29:03.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:29:03.546 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:29:03.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8afcd9ad-300d-4496-a56e-c391c0df221c 2024-04-25T08:29:03.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8afcd9ad-300d-4496-a56e-c391c0df221c' 2024-04-25T08:29:03.547 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 8afcd9ad-300d-4496-a56e-c391c0df221c 2024-04-25T08:29:03.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:29:03.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPFCpmSX2/IRAAwhgC7bhGOocOXZWHXC8kog== 2024-04-25T08:29:03.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPFCpmSX2/IRAAwhgC7bhGOocOXZWHXC8kog=="}' 2024-04-25T08:29:03.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8afcd9ad-300d-4496-a56e-c391c0df221c -i td/osd-recovery-stats/2/new.json 2024-04-25T08:29:04.018 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:29:04.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:29:04.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPFCpmSX2/IRAAwhgC7bhGOocOXZWHXC8kog== --osd-uuid 8afcd9ad-300d-4496-a56e-c391c0df221c 2024-04-25T08:29:04.068 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:04.071+0000 7fd16bb0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:04.105 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:04.107+0000 7fd16bb0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:04.110 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:04.111+0000 7fd16bb0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:04.110 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:04.111+0000 7fd16bb0d7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:29:04.111 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:04.111+0000 7fd16bb0d7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:29:04.111 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:04.111+0000 7fd16bb0d7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:29:04.152 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:04.155+0000 7fd16bb0d7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:29:11.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:29:11.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:29:11.114 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T08:29:11.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:29:11.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:29:11.704 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T08:29:11.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:29:11.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:29:11.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:29:11.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:29:11.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:29:11.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:11.731+0000 7fceb1f1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:11.735+0000 7fceb1f1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:11.733 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:11.735+0000 7fceb1f1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:12.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:29:12.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:29:12.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:29:12.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:29:12.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:29:12.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:12.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:29:12.180 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:29:12.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:12.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:12.295+0000 7fceb1f1a7c0 -1 Falling back to public interface 2024-04-25T08:29:12.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:29:13.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:13.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:14.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:14.921 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:14.923+0000 7fceb1f1a7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:29:15.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:15.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:15.047 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:29:15.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:29:15.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:15.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:15.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:16.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:16.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:16.770 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:29:16.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:29:16.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:16.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:17.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:18.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:18.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:18.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:29:18.244 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:29:18.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:18.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:18.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:19.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:19.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:19.707 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:29:19.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:29:19.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:19.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:20.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:21.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:21.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:21.273 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:29:21.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:29:21.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:21.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:21.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:22.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:22.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:22.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:29:22.765 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:29:22.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:22.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:23.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:24.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:24.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:24.260 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:29:24.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:29:24.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:24.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:24.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:25.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:25.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:25.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:29:25.686 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:29:25.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:25.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:26.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:27.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:27.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:27.149 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:29:27.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:29:27.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:27.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:27.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:28.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:28.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:28.748 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:29:28.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:29:28.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:28.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:29.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:30.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:30.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:30.197 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:29:30.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:29:30.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:30.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:31.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:32.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:32.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:32.251 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:29:32.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:29:32.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:32.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:32.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:33.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:33.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:33.869 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:29:33.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:29:33.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:33.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:34.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:35.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:35.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:35.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:29:35.706 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:29:35.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:35.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:36.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:37.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:37.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:37.283 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:29:37.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:29:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:37.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:38.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:39.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:39.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:39.379 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:29:39.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:29:39.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:39.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:40.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:41.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:41.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:41.523 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:29:41.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:29:41.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:41.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:29:41.945 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/285319757,v1:127.0.0.1:6819/285319757] [v2:127.0.0.1:6820/285319757,v1:127.0.0.1:6821/285319757] exists,up 8afcd9ad-300d-4496-a56e-c391c0df221c 2024-04-25T08:29:41.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:29:41.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:29:41.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:29:41.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-04-25T08:29:41.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:29:41.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:29:41.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:29:41.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:29:41.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:29:41.950 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:29:41.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:29:41.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:29:41.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2768dbc4-261f-4bca-8fdd-f32cdc6d81df 2024-04-25T08:29:41.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2768dbc4-261f-4bca-8fdd-f32cdc6d81df' 2024-04-25T08:29:41.952 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 2768dbc4-261f-4bca-8fdd-f32cdc6d81df 2024-04-25T08:29:41.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:29:41.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1FCpmDFYxOhAAUGM8uEJLr8JFT1vUbwhs3A== 2024-04-25T08:29:41.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1FCpmDFYxOhAAUGM8uEJLr8JFT1vUbwhs3A=="}' 2024-04-25T08:29:41.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2768dbc4-261f-4bca-8fdd-f32cdc6d81df -i td/osd-recovery-stats/3/new.json 2024-04-25T08:29:42.577 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:29:42.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:29:42.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1FCpmDFYxOhAAUGM8uEJLr8JFT1vUbwhs3A== --osd-uuid 2768dbc4-261f-4bca-8fdd-f32cdc6d81df 2024-04-25T08:29:42.623 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:42.626+0000 7f6891b127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:42.651 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:42.654+0000 7f6891b127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:42.653 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:42.654+0000 7f6891b127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:42.653 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:42.654+0000 7f6891b127c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:29:42.653 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:42.654+0000 7f6891b127c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:29:42.654 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:42.654+0000 7f6891b127c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:29:42.690 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:42.690+0000 7f6891b127c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:29:49.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:29:49.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:29:49.705 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T08:29:49.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:29:49.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:29:50.259 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T08:29:50.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:29:50.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:29:50.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:29:50.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:29:50.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:29:50.290 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:50.290+0000 7f83133f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:50.292 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:50.294+0000 7f83133f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:50.293 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:50.294+0000 7f83133f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:29:50.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:29:50.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:29:50.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:29:50.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:29:50.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:29:50.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:50.696 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:29:50.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:29:50.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:50.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:29:50.852 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:50.854+0000 7f83133f97c0 -1 Falling back to public interface 2024-04-25T08:29:51.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:52.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:52.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:52.154 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:29:52.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:29:52.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:52.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:29:52.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:53.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:53.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:53.602 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:29:53.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:29:53.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:53.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:29:53.632 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:29:53.634+0000 7f83133f97c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:29:54.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:55.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:55.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:55.026 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:29:55.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:29:55.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:55.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:29:55.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:56.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:56.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:56.603 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:29:56.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:29:56.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:56.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:29:57.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:58.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:58.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:58.122 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:29:58.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:29:58.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:58.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:29:58.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:29:59.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:29:59.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:29:59.605 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:29:59.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:29:59.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:29:59.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:00.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:01.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:01.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:01.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:30:01.060 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:30:01.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:01.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:01.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:02.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:02.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:02.621 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:30:02.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:30:02.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:02.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:03.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:04.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:04.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:04.147 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:30:04.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:30:04.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:04.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:04.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:05.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:05.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:05.756 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:30:05.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:30:05.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:05.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:06.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:07.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:07.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:07.206 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:30:07.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:30:07.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:07.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:07.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:08.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:30:08.737 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:30:08.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:08.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:09.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:10.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:10.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:10.370 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:30:10.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:30:10.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:10.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:11.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:12.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:12.360 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:30:12.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:12.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:30:12.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:12.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:13.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:14.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:14.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:14.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:30:14.339 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:30:14.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:14.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:15.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:16.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:16.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:16.183 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:30:16.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:30:16.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:16.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:17.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:18.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:18.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:18.351 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:30:18.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:30:18.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:18.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:19.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:20.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:20.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:20.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:30:20.617 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:30:20.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:20.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:30:21.043 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/565642466,v1:127.0.0.1:6827/565642466] [v2:127.0.0.1:6828/565642466,v1:127.0.0.1:6829/565642466] exists,up 2768dbc4-261f-4bca-8fdd-f32cdc6d81df 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:30:21.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:30:21.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:30:21.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:30:21.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:30:21.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:30:21.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:30:21.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:30:21.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:30:21.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:30:21.047 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:30:21.047 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:30:21.047 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:30:21.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:30:21.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:30:21.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb04390d-628b-4dab-8946-57d05bfd0ba3 2024-04-25T08:30:21.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 eb04390d-628b-4dab-8946-57d05bfd0ba3' 2024-04-25T08:30:21.051 INFO:tasks.workunit.client.0.smithi102.stdout:add osd4 eb04390d-628b-4dab-8946-57d05bfd0ba3 2024-04-25T08:30:21.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:30:21.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdFCpmTZaSBBAAFyMGcfiBqfMxkLKH2SEyyA== 2024-04-25T08:30:21.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdFCpmTZaSBBAAFyMGcfiBqfMxkLKH2SEyyA=="}' 2024-04-25T08:30:21.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb04390d-628b-4dab-8946-57d05bfd0ba3 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:30:21.541 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:30:21.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:30:21.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdFCpmTZaSBBAAFyMGcfiBqfMxkLKH2SEyyA== --osd-uuid eb04390d-628b-4dab-8946-57d05bfd0ba3 2024-04-25T08:30:21.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:21.594+0000 7f9440f007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:21.627 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:21.630+0000 7f9440f007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:21.629 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:21.630+0000 7f9440f007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:21.629 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:21.630+0000 7f9440f007c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:30:21.630 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:21.630+0000 7f9440f007c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:30:21.630 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:21.630+0000 7f9440f007c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:30:21.733 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:21.738+0000 7f9440f007c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:30:28.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:30:28.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:30:28.513 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd4 key to auth repository 2024-04-25T08:30:28.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:30:28.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:30:29.128 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.4 2024-04-25T08:30:29.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:30:29.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:30:29.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:30:29.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:30:29.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:30:29.159 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:29.161+0000 7f36f26467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:29.162 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:29.165+0000 7f36f26467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:29.163 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:29.165+0000 7f36f26467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:30:29.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:30:29.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:30:29.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:30:29.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:30:29.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:30:29.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:29.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:30:29.583 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:30:29.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:29.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:29.739 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:29.741+0000 7f36f26467c0 -1 Falling back to public interface 2024-04-25T08:30:30.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:31.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:31.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:31.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:30:31.012 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:30:31.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:31.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:31.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:32.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:32.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:32.417 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:30:32.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:30:32.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:32.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:32.424 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:30:32.425+0000 7f36f26467c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:30:33.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:34.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:34.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:34.016 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:30:34.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:30:34.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:34.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:34.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:35.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:35.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:35.579 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:30:35.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:30:35.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:35.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:36.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:37.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:37.039 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:30:37.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:37.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:30:37.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:37.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:37.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:38.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:38.507 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:30:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:30:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:38.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:38.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:39.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:39.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:39.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:30:39.937 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:30:39.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:39.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:40.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:41.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:41.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:41.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:30:41.584 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:30:41.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:41.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:42.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:43.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:43.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:43.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:30:43.181 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:30:43.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:43.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:43.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:44.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:44.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:44.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:30:44.762 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:30:44.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:44.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:45.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:46.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:46.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:46.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:30:46.380 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:30:46.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:46.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:46.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:47.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:47.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:47.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:30:47.854 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:30:47.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:47.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:48.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:49.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:49.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:49.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:30:49.753 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:30:49.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:49.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:50.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:51.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:51.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:51.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:30:51.586 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:30:51.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:51.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:52.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:53.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:53.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:53.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:30:53.483 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:30:53.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:53.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:54.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:55.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:55.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:55.342 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:30:55.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:30:55.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:55.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:56.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:57.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:57.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:57.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:30:57.200 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:30:57.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:57.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:58.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:30:59.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:30:59.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:30:59.020 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:30:59.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:30:59.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:30:59.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:30:59.511 INFO:tasks.workunit.client.0.smithi102.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2734239168,v1:127.0.0.1:6835/2734239168] [v2:127.0.0.1:6836/2734239168,v1:127.0.0.1:6837/2734239168] exists,up eb04390d-628b-4dab-8946-57d05bfd0ba3 2024-04-25T08:30:59.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:30:59.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:30:59.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:30:59.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-04-25T08:30:59.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:30:59.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:30:59.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:30:59.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:30:59.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:30:59.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:30:59.515 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:30:59.515 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:30:59.515 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:30:59.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:30:59.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:30:59.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:30:59.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1780b7fb-27a3-4e59-9703-f7079459d8be 2024-04-25T08:30:59.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 1780b7fb-27a3-4e59-9703-f7079459d8be' 2024-04-25T08:30:59.518 INFO:tasks.workunit.client.0.smithi102.stdout:add osd5 1780b7fb-27a3-4e59-9703-f7079459d8be 2024-04-25T08:30:59.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:30:59.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDFCpmAUY5IBAAxSgaosVcq9Za61duitmghA== 2024-04-25T08:30:59.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDFCpmAUY5IBAAxSgaosVcq9Za61duitmghA=="}' 2024-04-25T08:30:59.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1780b7fb-27a3-4e59-9703-f7079459d8be -i td/osd-recovery-stats/5/new.json 2024-04-25T08:31:00.076 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:31:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:31:00.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDFCpmAUY5IBAAxSgaosVcq9Za61duitmghA== --osd-uuid 1780b7fb-27a3-4e59-9703-f7079459d8be 2024-04-25T08:31:00.125 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:00.125+0000 7f8f1c3fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:00.184 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:00.185+0000 7f8f1c3fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:00.186 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:00.189+0000 7f8f1c3fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:00.187 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:00.189+0000 7f8f1c3fe7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:31:00.187 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:00.189+0000 7f8f1c3fe7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:31:00.187 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:00.189+0000 7f8f1c3fe7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:31:00.255 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:00.261+0000 7f8f1c3fe7c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:31:07.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:31:07.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:31:07.286 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd5 key to auth repository 2024-04-25T08:31:07.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:31:07.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:31:07.842 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.5 2024-04-25T08:31:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:31:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:31:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:31:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:31:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:31:07.867 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:07.869+0000 7fc08d64f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:07.901+0000 7fc08d64f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:07.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:07.905+0000 7fc08d64f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:31:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:31:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:31:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:31:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:31:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:31:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:31:08.275 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:31:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:08.475 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:08.477+0000 7fc08d64f7c0 -1 Falling back to public interface 2024-04-25T08:31:08.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:09.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:09.699 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:31:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:31:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:09.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:10.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:11.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:11.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:11.103 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:31:11.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:31:11.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:11.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:11.193 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:31:11.193+0000 7fc08d64f7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:31:11.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:12.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:12.622 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:31:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:31:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:12.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:13.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:14.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:14.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:14.303 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:31:14.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:31:14.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:14.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:14.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:15.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:15.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:15.855 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:31:15.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:31:15.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:15.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:16.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:17.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:17.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:17.462 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:31:17.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:31:17.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:17.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:18.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:19.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:19.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:19.021 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:31:19.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:31:19.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:19.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:19.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:20.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:20.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:20.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:31:20.724 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:31:20.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:20.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:21.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:22.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:22.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:22.293 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:31:22.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:31:22.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:22.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:22.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:23.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:23.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:23.728 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:31:23.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:31:23.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:23.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:24.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:25.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:25.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:25.210 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:31:25.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:31:25.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:25.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:25.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:26.778 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:31:26.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:26.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:26.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:31:26.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:26.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:27.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:28.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:28.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:28.867 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:31:28.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:31:28.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:28.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:29.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:30.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:30.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:30.796 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:31:30.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:31:30.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:30.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:31.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:32.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:32.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:32.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:31:32.316 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:31:32.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:32.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:33.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:34.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:34.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:34.247 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:31:34.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:31:34.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:34.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:35.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:36.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:36.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:36.030 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:31:36.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:31:36.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:36.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:36.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:37.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:37.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:37.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:31:37.754 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:31:37.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:37.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:38.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:31:39.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:31:39.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:31:39.523 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:31:39.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:31:39.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:31:39.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:31:39.945 INFO:tasks.workunit.client.0.smithi102.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3196150875,v1:127.0.0.1:6843/3196150875] [v2:127.0.0.1:6844/3196150875,v1:127.0.0.1:6845/3196150875] exists,up 1780b7fb-27a3-4e59-9703-f7079459d8be 2024-04-25T08:31:39.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:31:39.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:31:39.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:31:39.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' erasure = erasure ']' 2024-04-25T08:31:39.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:132: do_recovery_out1: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T08:31:40.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:133: do_recovery_out1: create_pool test 1 1 erasure myprofile 2024-04-25T08:31:40.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T08:31:41.487 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T08:31:41.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:31:42.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-04-25T08:31:42.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:31:42.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:31:42.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:31:42.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:31:42.510 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:31:42.511 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:31:42.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:31:42.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:31:42.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:31:42.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:31:42.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:31:42.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:31:42.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:31:42.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:31:42.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:31:43.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:31:43.107 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:31:43.107 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:31:43.107 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:31:43.107 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T08:31:43.107 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T08:31:43.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:31:43.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:43.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:31:43.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836527 2024-04-25T08:31:43.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836527 2024-04-25T08:31:43.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527' 2024-04-25T08:31:43.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:43.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:31:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673001 2024-04-25T08:31:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673001 2024-04-25T08:31:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001' 2024-04-25T08:31:43.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:43.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:31:43.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509468 2024-04-25T08:31:43.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509468 2024-04-25T08:31:43.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509468' 2024-04-25T08:31:43.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:43.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:31:43.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345947 2024-04-25T08:31:43.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345947 2024-04-25T08:31:43.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509468 3-85899345947' 2024-04-25T08:31:43.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:43.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:31:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182417 2024-04-25T08:31:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182417 2024-04-25T08:31:43.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509468 3-85899345947 4-107374182417' 2024-04-25T08:31:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:31:43.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:31:44.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018891 2024-04-25T08:31:44.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018891 2024-04-25T08:31:44.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949673001 2-64424509468 3-85899345947 4-107374182417 5-128849018891' 2024-04-25T08:31:44.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:44.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836527 2024-04-25T08:31:44.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:44.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:31:44.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836527 2024-04-25T08:31:44.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:44.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836527 2024-04-25T08:31:44.150 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836527 2024-04-25T08:31:44.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-04-25T08:31:44.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:31:44.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836525 -lt 21474836527 2024-04-25T08:31:44.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:31:45.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:31:45.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:31:45.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836528 -lt 21474836527 2024-04-25T08:31:45.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:45.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673001 2024-04-25T08:31:45.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:45.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:31:45.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673001 2024-04-25T08:31:45.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:45.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673001 2024-04-25T08:31:45.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-04-25T08:31:45.965 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673001 2024-04-25T08:31:45.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:31:46.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673001 -lt 42949673001 2024-04-25T08:31:46.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:46.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509468 2024-04-25T08:31:46.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:46.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:31:46.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509468 2024-04-25T08:31:46.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:46.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509468 2024-04-25T08:31:46.382 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509468 2024-04-25T08:31:46.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-04-25T08:31:46.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:31:46.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509468 -lt 64424509468 2024-04-25T08:31:46.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:46.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345947 2024-04-25T08:31:46.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:46.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:31:46.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:46.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345947 2024-04-25T08:31:46.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345947 2024-04-25T08:31:46.811 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345947 2024-04-25T08:31:46.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345947' 2024-04-25T08:31:46.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:31:47.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345948 -lt 85899345947 2024-04-25T08:31:47.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:47.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182417 2024-04-25T08:31:47.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:47.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:31:47.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182417 2024-04-25T08:31:47.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:47.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182417 2024-04-25T08:31:47.296 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182417 2024-04-25T08:31:47.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2024-04-25T08:31:47.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:31:47.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182418 -lt 107374182417 2024-04-25T08:31:47.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:31:47.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018891 2024-04-25T08:31:47.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:31:47.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:31:47.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018891 2024-04-25T08:31:47.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:31:47.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018891 2024-04-25T08:31:47.800 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018891 2024-04-25T08:31:47.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018891' 2024-04-25T08:31:47.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:31:48.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018892 -lt 128849018891 2024-04-25T08:31:48.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:31:48.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:48.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:48.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:31:48.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:48.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:48.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:48.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:48.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:48.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:48.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:49.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:49.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:49.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:49.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:49.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:31:49.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:31:49.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:31:49.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-04-25T08:31:49.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:49.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-04-25T08:31:49.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:49.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-04-25T08:31:50.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:50.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-04-25T08:31:50.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:50.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-04-25T08:31:50.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:50.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-04-25T08:31:50.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:50.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-04-25T08:31:51.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:51.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-04-25T08:31:51.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:51.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-04-25T08:31:51.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:51.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-04-25T08:31:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:51.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-04-25T08:31:52.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:52.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-04-25T08:31:52.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:52.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-04-25T08:31:52.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:52.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-04-25T08:31:52.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:52.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-04-25T08:31:52.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:52.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-04-25T08:31:53.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:53.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-04-25T08:31:53.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:53.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-04-25T08:31:53.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:53.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-04-25T08:31:53.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:53.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-04-25T08:31:54.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:54.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-04-25T08:31:54.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:54.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-04-25T08:31:54.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:54.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-04-25T08:31:54.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:54.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-04-25T08:31:55.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:55.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-04-25T08:31:55.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:55.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-04-25T08:31:55.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:55.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-04-25T08:31:55.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:55.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-04-25T08:31:55.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:55.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-04-25T08:31:56.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:56.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-04-25T08:31:56.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:56.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-04-25T08:31:56.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:56.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-04-25T08:31:56.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:56.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-04-25T08:31:57.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:57.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-04-25T08:31:57.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:57.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-04-25T08:31:57.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:57.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-04-25T08:31:57.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-04-25T08:31:58.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:58.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-04-25T08:31:58.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:58.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-04-25T08:31:58.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:58.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-04-25T08:31:58.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:58.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-04-25T08:31:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:59.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-04-25T08:31:59.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:59.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-04-25T08:31:59.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:59.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-04-25T08:31:59.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:31:59.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-04-25T08:32:00.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:00.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-04-25T08:32:00.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:00.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-04-25T08:32:00.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:00.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-04-25T08:32:00.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:00.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-04-25T08:32:00.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:00.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-04-25T08:32:01.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:01.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-04-25T08:32:01.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:01.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-04-25T08:32:01.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:01.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-04-25T08:32:01.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:01.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-04-25T08:32:02.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:02.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-04-25T08:32:02.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:02.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-04-25T08:32:02.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:02.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-04-25T08:32:02.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:02.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-04-25T08:32:02.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:02.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-04-25T08:32:03.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:03.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-04-25T08:32:03.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:03.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-04-25T08:32:03.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:03.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-04-25T08:32:03.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:03.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-04-25T08:32:04.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:04.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-04-25T08:32:04.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:04.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-04-25T08:32:04.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:04.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-04-25T08:32:04.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:04.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-04-25T08:32:05.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:05.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-04-25T08:32:05.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:05.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-04-25T08:32:05.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:05.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-04-25T08:32:05.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:05.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-04-25T08:32:06.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:06.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-04-25T08:32:06.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:06.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-04-25T08:32:06.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:06.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-04-25T08:32:06.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:06.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-04-25T08:32:06.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:06.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-04-25T08:32:07.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:07.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-04-25T08:32:07.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:07.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-04-25T08:32:07.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:07.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-04-25T08:32:07.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:07.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-04-25T08:32:08.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:08.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-04-25T08:32:08.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:08.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-04-25T08:32:08.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:08.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-04-25T08:32:08.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:08.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-04-25T08:32:09.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:09.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-04-25T08:32:09.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:09.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-04-25T08:32:09.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:09.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-04-25T08:32:09.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:09.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-04-25T08:32:09.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:09.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-04-25T08:32:10.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:10.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-04-25T08:32:10.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:10.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-04-25T08:32:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:10.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-04-25T08:32:11.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:11.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-04-25T08:32:11.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:11.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-04-25T08:32:11.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:11.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-04-25T08:32:11.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:11.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-04-25T08:32:11.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:11.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-04-25T08:32:12.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:12.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-04-25T08:32:12.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:12.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-04-25T08:32:12.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:12.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-04-25T08:32:12.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:12.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-04-25T08:32:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:13.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-04-25T08:32:13.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:13.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-04-25T08:32:13.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:13.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-04-25T08:32:13.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:13.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-04-25T08:32:14.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:14.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-04-25T08:32:14.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:14.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-04-25T08:32:14.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:14.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-04-25T08:32:15.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:15.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-04-25T08:32:15.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:15.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-04-25T08:32:15.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:15.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-04-25T08:32:15.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:15.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-04-25T08:32:15.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:15.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-04-25T08:32:16.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:16.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-04-25T08:32:16.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:16.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-04-25T08:32:16.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:16.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-04-25T08:32:16.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:16.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-04-25T08:32:17.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:17.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-04-25T08:32:17.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:17.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-04-25T08:32:17.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:17.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-04-25T08:32:17.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:17.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-04-25T08:32:18.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:18.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-04-25T08:32:18.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:18.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-04-25T08:32:18.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:18.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-04-25T08:32:18.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:18.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-04-25T08:32:19.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:19.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-04-25T08:32:19.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:19.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-04-25T08:32:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-04-25T08:32:19.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:19.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-04-25T08:32:20.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:20.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-04-25T08:32:20.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:20.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-04-25T08:32:20.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:20.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-04-25T08:32:20.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:20.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-04-25T08:32:20.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:20.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-04-25T08:32:21.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:21.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-04-25T08:32:21.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:21.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-04-25T08:32:21.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:21.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-04-25T08:32:21.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:21.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-04-25T08:32:22.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:22.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-04-25T08:32:22.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:22.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-04-25T08:32:22.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:22.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-04-25T08:32:22.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:22.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-04-25T08:32:23.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:23.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-04-25T08:32:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:23.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-04-25T08:32:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:23.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-04-25T08:32:23.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:23.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-04-25T08:32:24.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:24.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-04-25T08:32:24.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:24.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-04-25T08:32:24.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:24.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-04-25T08:32:24.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:24.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-04-25T08:32:24.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:24.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-04-25T08:32:25.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:25.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-04-25T08:32:25.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:25.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-04-25T08:32:25.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:25.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-04-25T08:32:25.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:25.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-04-25T08:32:26.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:26.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-04-25T08:32:26.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:26.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-04-25T08:32:26.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:26.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-04-25T08:32:26.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:26.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-04-25T08:32:27.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:27.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-04-25T08:32:27.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:27.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-04-25T08:32:27.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:27.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-04-25T08:32:27.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:27.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-04-25T08:32:28.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:28.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-04-25T08:32:28.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:28.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-04-25T08:32:28.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:28.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-04-25T08:32:28.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:28.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-04-25T08:32:28.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:28.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-04-25T08:32:29.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:29.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-04-25T08:32:29.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:29.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-04-25T08:32:29.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:29.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-04-25T08:32:29.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:29.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-04-25T08:32:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:30.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-04-25T08:32:30.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:30.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-04-25T08:32:30.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:30.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-04-25T08:32:30.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:30.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-04-25T08:32:30.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:30.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-04-25T08:32:31.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:31.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-04-25T08:32:31.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:31.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-04-25T08:32:31.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:31.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-04-25T08:32:31.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:31.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-04-25T08:32:32.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:32.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-04-25T08:32:32.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:32.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-04-25T08:32:32.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:32.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-04-25T08:32:32.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:32.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-04-25T08:32:32.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:32.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-04-25T08:32:33.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:33.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-04-25T08:32:33.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:33.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-04-25T08:32:33.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:33.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-04-25T08:32:33.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:33.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-04-25T08:32:34.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:34.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-04-25T08:32:34.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:34.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-04-25T08:32:34.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:34.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-04-25T08:32:34.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:34.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-04-25T08:32:35.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:35.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-04-25T08:32:35.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:35.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-04-25T08:32:35.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:35.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-04-25T08:32:35.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:35.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-04-25T08:32:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:36.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-04-25T08:32:36.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:36.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-04-25T08:32:36.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:32:36.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-04-25T08:32:36.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-04-25T08:32:36.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:32:36.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:32:36.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:32:36.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:32:37.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-04-25T08:32:37.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-04-25T08:32:37.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:32:37.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:32:37.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:32:37.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:32:37.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-04-25T08:32:37.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-04-25T08:32:37.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:32:37.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:32:37.604 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:32:37.604 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:32:37.605 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:32:37.605 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:32:37.605 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:32:38.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:32:38.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:32:38.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:32:38.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-04-25T08:32:38.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-04-25T08:32:39.230 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is set 2024-04-25T08:32:39.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:32:39.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 278597 2024-04-25T08:32:39.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-04-25T08:32:39.904 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T08:32:39.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-04-25T08:32:40.834 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. 2024-04-25T08:32:40.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-04-25T08:32:42.507 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is unset 2024-04-25T08:32:42.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-04-25T08:32:42.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:32:42.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:32:42.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:32:42.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:32:43.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:32:43.361 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:32:43.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-04-25T08:32:45.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-04-25T08:32:45.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:32:45.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:32:45.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:32:45.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:32:45.380 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:32:45.380 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:32:45.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:32:45.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:32:45.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:32:45.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:32:45.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:32:45.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:32:45.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:32:45.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:32:45.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:32:45.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:32:45.975 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:32:45.975 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:32:45.975 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:32:45.975 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T08:32:45.975 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T08:32:45.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:32:45.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:32:45.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:32:46.118 INFO:tasks.workunit.client.0.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:32:46.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:32:46.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:32:46.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:32:46.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:32:46.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:32:46.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673015 2024-04-25T08:32:46.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673015 2024-04-25T08:32:46.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673015' 2024-04-25T08:32:46.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:32:46.278 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:32:46.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509482 2024-04-25T08:32:46.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509482 2024-04-25T08:32:46.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673015 2-64424509482' 2024-04-25T08:32:46.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:32:46.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:32:46.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345962 2024-04-25T08:32:46.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345962 2024-04-25T08:32:46.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673015 2-64424509482 3-85899345962' 2024-04-25T08:32:46.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:32:46.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:32:46.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182432 2024-04-25T08:32:46.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182432 2024-04-25T08:32:46.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673015 2-64424509482 3-85899345962 4-107374182432' 2024-04-25T08:32:46.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:32:46.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:32:46.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018906 2024-04-25T08:32:46.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018906 2024-04-25T08:32:46.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673015 2-64424509482 3-85899345962 4-107374182432 5-128849018906' 2024-04-25T08:32:46.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:32:46.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673015 2024-04-25T08:32:46.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:32:46.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:32:46.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673015 2024-04-25T08:32:46.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:32:46.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673015 2024-04-25T08:32:46.879 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673015 2024-04-25T08:32:46.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673015' 2024-04-25T08:32:46.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:32:47.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673015 -lt 42949673015 2024-04-25T08:32:47.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:32:47.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509482 2024-04-25T08:32:47.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:32:47.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:32:47.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509482 2024-04-25T08:32:47.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:32:47.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509482 2024-04-25T08:32:47.309 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509482 2024-04-25T08:32:47.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-04-25T08:32:47.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:32:47.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509482 -lt 64424509482 2024-04-25T08:32:47.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:32:47.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345962 2024-04-25T08:32:47.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:32:47.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:32:47.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345962 2024-04-25T08:32:47.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:32:47.905 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345962 2024-04-25T08:32:47.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345962 2024-04-25T08:32:47.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345962' 2024-04-25T08:32:47.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:32:48.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345962 -lt 85899345962 2024-04-25T08:32:48.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:32:48.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182432 2024-04-25T08:32:48.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:32:48.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:32:48.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182432 2024-04-25T08:32:48.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:32:48.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182432 2024-04-25T08:32:48.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182432' 2024-04-25T08:32:48.309 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182432 2024-04-25T08:32:48.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:32:48.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182432 -lt 107374182432 2024-04-25T08:32:48.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:32:48.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018906 2024-04-25T08:32:48.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:32:48.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:32:48.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:32:48.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018906 2024-04-25T08:32:48.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018906 2024-04-25T08:32:48.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018906' 2024-04-25T08:32:48.918 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018906 2024-04-25T08:32:48.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:32:49.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018906 -lt 128849018906 2024-04-25T08:32:49.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:32:49.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:49.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:49.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:32:49.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:49.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:49.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:49.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:49.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:49.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:49.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:50.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:50.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:50.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:50.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:50.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:50.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:32:50.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:32:50.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:32:50.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:32:50.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:51.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:51.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:51.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:51.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:51.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:51.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:51.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:51.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:51.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:51.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:51.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:51.600 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:52.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:52.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:32:52.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:52.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:52.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:52.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:52.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:52.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:52.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:52.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:32:52.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:32:52.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:52.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:32:52.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:52.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:52.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:52.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:52.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:52.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:52.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:52.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:52.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:53.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:53.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:53.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:53.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:53.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:53.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:32:53.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:53.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:53.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:53.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:53.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:53.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:53.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:54.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12 2024-04-25T08:32:54.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12 '!=' null 2024-04-25T08:32:54.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:54.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:32:54.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:54.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:54.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:54.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:54.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:54.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:54.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:54.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:54.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:55.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:55.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:55.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:55.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:55.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:55.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:32:55.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:55.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:55.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:55.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:55.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:55.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:55.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:56.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-04-25T08:32:56.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-04-25T08:32:56.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:56.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:32:56.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:56.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:56.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:56.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:56.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:56.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:56.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:56.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:56.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:56.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:56.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:56.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:56.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:57.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:57.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:32:57.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:57.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:57.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:57.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:57.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:57.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:57.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:58.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-04-25T08:32:58.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-04-25T08:32:58.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:58.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:32:58.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:58.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:58.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:58.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:58.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:58.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:58.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:58.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:58.178 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:58.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:32:58.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:58.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:58.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:32:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:32:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:59.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:59.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:59.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:59.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:59.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:00.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12 2024-04-25T08:33:00.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12 '!=' null 2024-04-25T08:33:00.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:00.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:33:00.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:00.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:00.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:00.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:00.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:00.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:00.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:00.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:00.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:00.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:33:00.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:00.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:00.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:01.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:33:01.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:33:01.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:01.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:01.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:01.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:01.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:01.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:01.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:01.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:33:01.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:33:01.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:01.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:33:01.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:01.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:01.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:01.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:01.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:01.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:01.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:01.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:01.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:02.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:33:02.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:02.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:02.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:33:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:33:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:02.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:02.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:02.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:02.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:02.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:02.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:03.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:33:03.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:33:03.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:03.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:33:03.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:03.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:03.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:03.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:03.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:03.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:03.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:03.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:03.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:03.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:33:03.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:03.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:03.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:04.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:33:04.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:33:04.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:33:04.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 erasure 200 0 0 0 2024-04-25T08:33:04.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:33:04.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:33:04.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:33:04.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:33:04.297 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:33:04.298 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:33:04.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,2]/[1,NONE,2]' 2024-04-25T08:33:04.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:33:04.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:33:04.318 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:33:04.318 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-04-25T08:33:04.319 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:33:04.319 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:33:04.319 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:33:04.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:33:04.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:33:04.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:33:04.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:33:04.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:33:04.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:33:04.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:33:04.327 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:33:04.327 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:33:04.327 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:33:04.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:33:04.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:33:04.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:33:04.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:33:04.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:33:04.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:33:04.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:33:04.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:33:04.339 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:33:04.340 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-04-25T08:33:04.340 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:33:04.340 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:33:04.340 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:33:04.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T08:33:04.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T08:33:04.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:33:04.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:33:04.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:33:04.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:33:04.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:33:04.347 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:33:04.347 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:33:04.347 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:33:04.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:33:04.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:33:04.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:33:04.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:33:04.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:33:04.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:33:04.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:33:04.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-04-25T08:33:04.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:33:04.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:33:04.956 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T08:33:04.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-04-25T08:33:04.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:33:04.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:33:04.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:33:04.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:33:04.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:33:06.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:33:06.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:33:06.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:33:06.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:33:06.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:33:06.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:33:06.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:33:06.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:33:06.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:33:06.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:33:06.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:33:06.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:33:06.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:33:06.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:33:06.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:33:06.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:33:06.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:33:06.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:33:06.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:33:06.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:33:06.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:33:06.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:33:06.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:33:06.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:33:06.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:33:06.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:06.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:33:06.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:33:06.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:33:06.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:33:06.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:33:06.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:33:06.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:33:06.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:33:06.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:33:06.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:33:06.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:33:06.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:33:06.345 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:33:06.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:33:06.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:33:06.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:33:06.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:33:06.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:33:06.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:33:06.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:33:06.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:33:06.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:33:06.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:33:06.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:33:06.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:33:06.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:33:06.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:33:06.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:33:06.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:33:06.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:33:06.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:33:06.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:06.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:33:06.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:33:06.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:33:06.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:33:06.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:33:06.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:33:06.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:06.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:33:06.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.277516 2024-04-25T08:33:06.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:33:06.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:33:06.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:33:06.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:33:06.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_remapped td/osd-recovery-stats 2024-04-25T08:33:06.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:384: TEST_recovery_erasure_remapped: local dir=td/osd-recovery-stats 2024-04-25T08:33:06.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:386: TEST_recovery_erasure_remapped: run_mon td/osd-recovery-stats a 2024-04-25T08:33:06.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:33:06.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:33:06.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:33:06.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:33:06.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:33:06.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:33:06.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:33:06.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:33:06.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:33:06.853 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:33:06.853 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:06.853 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:33:06.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:33:06.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:33:07.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:33:07.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:33:07.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:33:07.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:33:07.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:33:07.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:33:07.083 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:33:07.083 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:33:07.083 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:33:07.083 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:33:07.083 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:07.084 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:33:07.084 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:33:07.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:33:07.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get fsid 2024-04-25T08:33:07.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:33:07.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:33:07.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:33:07.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:33:07.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:33:07.205 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:33:07.205 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:33:07.205 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:33:07.205 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:33:07.205 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:07.205 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:33:07.205 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:33:07.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:33:07.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get mon_host 2024-04-25T08:33:07.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:387: TEST_recovery_erasure_remapped: run_mgr td/osd-recovery-stats x 2024-04-25T08:33:07.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:33:07.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:33:07.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:33:07.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:33:07.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:33:07.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:33:08.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:33:08.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:33:08.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:33:08.235 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:33:08.235 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:08.235 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:33:08.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:33:08.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:33:08.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:388: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 0 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:33:08.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:33:08.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:33:08.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:33:08.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:33:08.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:33:08.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:33:08.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:33:08.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:33:08.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:33:08.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:33:08.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:33:08.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:33:08.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=597e92c1-09a0-4902-8813-12e94ee4d1ec 2024-04-25T08:33:08.277 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 597e92c1-09a0-4902-8813-12e94ee4d1ec 2024-04-25T08:33:08.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 597e92c1-09a0-4902-8813-12e94ee4d1ec' 2024-04-25T08:33:08.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:33:08.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEFSpmUH3UERAAlXJswxpvv2JMOLX2e9SmSA== 2024-04-25T08:33:08.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEFSpmUH3UERAAlXJswxpvv2JMOLX2e9SmSA=="}' 2024-04-25T08:33:08.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 597e92c1-09a0-4902-8813-12e94ee4d1ec -i td/osd-recovery-stats/0/new.json 2024-04-25T08:33:08.625 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:33:08.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:33:08.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEFSpmUH3UERAAlXJswxpvv2JMOLX2e9SmSA== --osd-uuid 597e92c1-09a0-4902-8813-12e94ee4d1ec 2024-04-25T08:33:08.683 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:08.686+0000 7f39fcca27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:08.728 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:08.730+0000 7f39fcca27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:08.729 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:08.734+0000 7f39fcca27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:08.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:08.734+0000 7f39fcca27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:33:08.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:08.734+0000 7f39fcca27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:33:08.730 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:08.734+0000 7f39fcca27c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:33:08.775 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:08.778+0000 7f39fcca27c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:33:15.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:33:15.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:33:15.293 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:33:15.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:33:15.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:33:15.836 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:33:15.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:33:15.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:33:15.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:33:15.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:33:15.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:33:15.867 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:15.870+0000 7f8ea33b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:15.869 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:15.874+0000 7f8ea33b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:15.871 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:15.874+0000 7f8ea33b07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:16.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:33:16.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:33:16.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:33:16.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:33:16.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:33:16.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:16.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:33:16.248 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:33:16.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:16.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:16.454 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:16.458+0000 7f8ea33b07c0 -1 Falling back to public interface 2024-04-25T08:33:16.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:17.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:17.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:17.688 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:33:17.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:33:17.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:17.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:18.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:19.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:19.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:19.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:33:19.100 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:33:19.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:19.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:19.101 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:19.106+0000 7f8ea33b07c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:33:19.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:20.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:20.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:20.632 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:33:20.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:33:20.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:20.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:21.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:22.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:22.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:22.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:33:22.138 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:33:22.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:22.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:22.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:23.606 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:33:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:33:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:24.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:25.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:25.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:25.187 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:33:25.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:33:25.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:25.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:25.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:26.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:26.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:26.685 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:33:26.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:33:26.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:26.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:27.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:28.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:28.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:28.278 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:33:28.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:33:28.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:28.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:28.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:29.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:29.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:29.746 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:33:29.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:33:29.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:29.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:30.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:31.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:31.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:31.344 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:33:31.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:33:31.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:31.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:31.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:32.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:32.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:32.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:33:32.789 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:33:32.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:32.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:33.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:34.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:34.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:34.649 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:33:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:33:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:34.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:35.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:36.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:36.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:36.561 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:33:36.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:33:36.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:36.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:37.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:38.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:38.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:38.326 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:33:38.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:33:38.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:38.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:39.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:40.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:40.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:40.052 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:33:40.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:33:40.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:40.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:40.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:41.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:41.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:41.551 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:33:41.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:33:41.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:41.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:42.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:43.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:43.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:43.618 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:33:43.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:33:43.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:43.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:44.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:45.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:45.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:45.500 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:33:45.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:33:45.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:45.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:45.918 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/33006870,v1:127.0.0.1:6803/33006870] [v2:127.0.0.1:6804/33006870,v1:127.0.0.1:6805/33006870] exists,up 597e92c1-09a0-4902-8813-12e94ee4d1ec 2024-04-25T08:33:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:33:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:33:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:33:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:389: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 1 2024-04-25T08:33:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:33:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:33:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:33:45.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:45.921 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:33:45.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:33:45.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:33:45.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:33:45.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:33:45.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:33:45.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:33:45.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:33:45.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:33:45.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:33:45.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:33:45.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:33:45.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:33:45.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:33:45.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:33:45.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:33:45.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c5fb5e5a-7e96-4939-b9da-00e4885a3c82 2024-04-25T08:33:45.924 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 c5fb5e5a-7e96-4939-b9da-00e4885a3c82 2024-04-25T08:33:45.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c5fb5e5a-7e96-4939-b9da-00e4885a3c82' 2024-04-25T08:33:45.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:33:45.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpFSpmSZBkOBAAjZpp0FgGZ8e3umNi2FHKlA== 2024-04-25T08:33:45.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpFSpmSZBkOBAAjZpp0FgGZ8e3umNi2FHKlA=="}' 2024-04-25T08:33:45.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c5fb5e5a-7e96-4939-b9da-00e4885a3c82 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:33:46.575 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:33:46.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:33:46.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpFSpmSZBkOBAAjZpp0FgGZ8e3umNi2FHKlA== --osd-uuid c5fb5e5a-7e96-4939-b9da-00e4885a3c82 2024-04-25T08:33:46.628 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:46.634+0000 7f2bb0d9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:46.630 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:46.634+0000 7f2bb0d9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:46.631 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:46.634+0000 7f2bb0d9a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:46.632 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:46.634+0000 7f2bb0d9a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:33:46.632 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:46.634+0000 7f2bb0d9a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:33:46.632 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:46.634+0000 7f2bb0d9a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:33:46.675 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:46.678+0000 7f2bb0d9a7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:33:53.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:33:53.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:33:53.358 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T08:33:53.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:33:53.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:33:53.895 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:33:53.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:33:53.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:33:53.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:33:53.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:33:53.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:33:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:53.929+0000 7f5bd14bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:53.958 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:53.961+0000 7f5bd14bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:53.960 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:53.961+0000 7f5bd14bb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:54.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:33:54.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:33:54.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:33:54.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:33:54.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:33:54.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:54.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:33:54.320 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:33:54.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:54.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:54.514 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:54.517+0000 7f5bd14bb7c0 -1 Falling back to public interface 2024-04-25T08:33:54.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:55.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:55.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:55.737 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:33:55.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:33:55.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:55.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:56.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:57.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:57.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:57.150 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:33:57.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:33:57.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:57.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:57.205 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:33:57.209+0000 7f5bd14bb7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:33:57.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:33:58.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:33:58.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:58.765 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:33:58.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:33:58.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:58.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:33:59.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:00.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:00.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:00.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:34:00.292 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:34:00.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:00.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:00.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:01.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:01.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:01.740 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:34:01.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:34:01.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:01.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:02.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:03.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:03.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:03.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:34:03.302 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:34:03.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:03.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:03.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:04.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:04.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:04.887 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:34:04.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:34:04.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:04.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:05.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:06.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:06.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:06.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:34:06.705 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:34:06.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:06.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:07.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:08.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:08.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:08.400 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:34:08.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:34:08.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:08.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:08.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:09.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:09.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:09.850 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:34:09.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:34:09.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:09.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:10.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:11.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:11.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:11.349 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:34:11.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:34:11.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:11.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:11.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:12.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:12.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:12.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:34:12.858 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:34:12.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:12.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:13.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:14.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:14.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:14.778 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:34:14.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:34:14.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:14.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:16.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:17.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:17.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:17.060 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:34:17.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:34:17.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:17.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:17.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:18.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:18.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:18.646 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:34:18.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:34:18.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:18.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:19.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:20.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:20.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:20.395 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:34:20.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:34:20.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:20.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:21.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:22.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:22.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:22.095 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:34:22.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:34:22.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:22.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:22.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:23.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:23.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:23.935 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:34:23.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:34:23.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:23.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:24.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:25.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:25.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:25.428 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:34:25.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:34:25.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:25.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:25.847 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/338137429,v1:127.0.0.1:6811/338137429] [v2:127.0.0.1:6812/338137429,v1:127.0.0.1:6813/338137429] exists,up c5fb5e5a-7e96-4939-b9da-00e4885a3c82 2024-04-25T08:34:25.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:390: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 2 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:34:25.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:34:25.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:34:25.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:34:25.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:34:25.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:34:25.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:34:25.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:34:25.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:34:25.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:34:25.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:34:25.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:34:25.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:34:25.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a481083f-c50a-4c7c-9ebf-ceef0b646438 2024-04-25T08:34:25.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a481083f-c50a-4c7c-9ebf-ceef0b646438' 2024-04-25T08:34:25.855 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 a481083f-c50a-4c7c-9ebf-ceef0b646438 2024-04-25T08:34:25.855 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:34:25.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRFSpmjvRCNBAAlCZw5vUj4xePulwqJermlw== 2024-04-25T08:34:25.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRFSpmjvRCNBAAlCZw5vUj4xePulwqJermlw=="}' 2024-04-25T08:34:25.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a481083f-c50a-4c7c-9ebf-ceef0b646438 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:34:26.348 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:34:26.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:34:26.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRFSpmjvRCNBAAlCZw5vUj4xePulwqJermlw== --osd-uuid a481083f-c50a-4c7c-9ebf-ceef0b646438 2024-04-25T08:34:26.398 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:26.401+0000 7f51433907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:26.435 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:26.437+0000 7f51433907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:26.436 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:26.441+0000 7f51433907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:26.437 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:26.441+0000 7f51433907c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:34:26.437 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:26.441+0000 7f51433907c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:34:26.437 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:26.441+0000 7f51433907c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:34:26.474 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:26.477+0000 7f51433907c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:34:33.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:34:33.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:34:33.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:34:33.704 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T08:34:33.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:34:34.248 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T08:34:34.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:34:34.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:34:34.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:34:34.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:34:34.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:34:34.283 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:34.285+0000 7f0c278a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:34.285 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:34.289+0000 7f0c278a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:34.286 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:34.289+0000 7f0c278a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:34.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:34:34.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:34:34.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:34:34.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:34:34.654 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:34:34.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:34:34.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:34.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:34:34.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:34.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:34.829 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:34.833+0000 7f0c278a17c0 -1 Falling back to public interface 2024-04-25T08:34:35.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:36.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:36.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:36.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:34:36.094 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:34:36.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:36.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:36.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:37.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:37.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:37.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:34:37.507 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:34:37.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:37.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:37.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:34:37.596+0000 7f0c278a17c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:34:38.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:39.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:39.013 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:34:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:34:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:39.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:39.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:40.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:40.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:40.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:34:40.576 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:34:40.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:40.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:41.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:42.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:42.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:42.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:34:42.158 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:34:42.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:42.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:42.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:43.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:43.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:43.738 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:34:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:34:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:43.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:44.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:45.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:45.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:45.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:34:45.218 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:34:45.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:45.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:45.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:46.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:46.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:46.690 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:34:46.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:34:46.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:46.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:47.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:48.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:48.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:48.152 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:34:48.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:34:48.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:48.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:48.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:49.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:49.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:49.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:34:49.675 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:34:49.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:49.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:50.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:51.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:51.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:51.165 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:34:51.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:34:51.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:51.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:51.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:52.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:52.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:52.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:34:52.712 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:34:52.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:52.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:53.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:54.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:54.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:54.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:34:54.158 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:34:54.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:54.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:54.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:55.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:55.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:55.937 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:34:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:34:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:55.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:56.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:57.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:57.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:57.823 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:34:57.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:34:57.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:57.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:34:58.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:59.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:59.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:59.598 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:34:59.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:34:59.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:59.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:35:00.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:01.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:01.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:01.258 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:35:01.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:35:01.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:01.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:35:01.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:02.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:02.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:02.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:35:02.888 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:35:02.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:02.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:35:03.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:04.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:04.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:04.954 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:35:04.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:35:04.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:04.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:35:05.374 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4069516697,v1:127.0.0.1:6819/4069516697] [v2:127.0.0.1:6820/4069516697,v1:127.0.0.1:6821/4069516697] exists,up a481083f-c50a-4c7c-9ebf-ceef0b646438 2024-04-25T08:35:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:35:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:35:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:35:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:391: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 3 2024-04-25T08:35:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:35:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:35:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:35:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:35:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:35:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:35:05.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:35:05.379 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:35:05.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:35:05.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:35:05.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:35:05.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:35:05.381 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 0cae5fc4-4fe1-49bd-bbda-7194fc11e24f 2024-04-25T08:35:05.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0cae5fc4-4fe1-49bd-bbda-7194fc11e24f 2024-04-25T08:35:05.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0cae5fc4-4fe1-49bd-bbda-7194fc11e24f' 2024-04-25T08:35:05.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:35:05.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC5FSpm5FddGBAAy30EaJx+ssMDXOoqh/K9lg== 2024-04-25T08:35:05.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC5FSpm5FddGBAAy30EaJx+ssMDXOoqh/K9lg=="}' 2024-04-25T08:35:05.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0cae5fc4-4fe1-49bd-bbda-7194fc11e24f -i td/osd-recovery-stats/3/new.json 2024-04-25T08:35:06.092 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:35:06.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:35:06.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC5FSpm5FddGBAAy30EaJx+ssMDXOoqh/K9lg== --osd-uuid 0cae5fc4-4fe1-49bd-bbda-7194fc11e24f 2024-04-25T08:35:06.149 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:06.152+0000 7f953dd907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:06.165 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:06.168+0000 7f953dd907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:06.166 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:06.168+0000 7f953dd907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:06.166 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:06.168+0000 7f953dd907c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:35:06.167 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:06.168+0000 7f953dd907c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:35:06.167 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:06.168+0000 7f953dd907c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:35:06.243 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:06.248+0000 7f953dd907c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:35:13.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:35:13.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:35:13.243 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T08:35:13.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:35:13.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:35:13.825 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T08:35:13.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:35:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:35:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:35:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:35:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:35:13.856 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:13.860+0000 7fa63ede17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:13.882 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:13.884+0000 7fa63ede17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:13.883 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:13.888+0000 7fa63ede17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:35:14.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:35:14.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:35:14.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:35:14.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:35:14.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:35:14.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:14.228 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:35:14.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:35:14.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:14.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:14.445 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:14.448+0000 7fa63ede17c0 -1 Falling back to public interface 2024-04-25T08:35:14.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:15.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:15.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:15.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:35:15.687 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:35:15.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:15.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:16.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:17.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:17.106 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:35:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:35:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:17.150 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:35:17.152+0000 7fa63ede17c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:35:17.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:18.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:18.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:18.724 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:35:18.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:35:18.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:18.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:19.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:20.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:20.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:20.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:35:20.402 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:35:20.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:20.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:20.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:21.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:21.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:21.890 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:35:21.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:35:21.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:21.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:22.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:23.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:23.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:23.335 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:35:23.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:35:23.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:23.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:23.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:24.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:24.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:24.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:35:24.839 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:35:24.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:24.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:25.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:26.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:26.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:26.613 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:35:26.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:35:26.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:26.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:27.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:28.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:28.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:28.204 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:35:28.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:35:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:28.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:28.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:29.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:29.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:29.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:35:29.704 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:35:29.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:29.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:30.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:31.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:31.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:31.283 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:35:31.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:35:31.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:31.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:31.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:32.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:32.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:32.763 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:35:32.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:35:32.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:32.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:34.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:34.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:34.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:35:34.631 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:35:34.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:34.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:35.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:36.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:36.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:36.472 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:35:36.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:35:36.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:36.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:37.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:38.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:38.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:38.308 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:35:38.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:35:38.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:38.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:39.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:40.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:40.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:40.270 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:35:40.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:35:40.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:40.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:41.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:42.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:42.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:42.053 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:35:42.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:35:42.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:42.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:42.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:44.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:44.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:44.002 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:35:44.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:35:44.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:44.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:35:44.470 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/730709919,v1:127.0.0.1:6827/730709919] [v2:127.0.0.1:6828/730709919,v1:127.0.0.1:6829/730709919] exists,up 0cae5fc4-4fe1-49bd-bbda-7194fc11e24f 2024-04-25T08:35:44.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:35:44.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:35:44.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:35:44.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:393: TEST_recovery_erasure_remapped: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T08:35:45.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:394: TEST_recovery_erasure_remapped: create_pool test 1 1 erasure myprofile 2024-04-25T08:35:45.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T08:35:46.014 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T08:35:46.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:35:47.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:395: TEST_recovery_erasure_remapped: ceph osd pool set test min_size 2 2024-04-25T08:35:47.945 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 min_size to 2 2024-04-25T08:35:47.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:397: TEST_recovery_erasure_remapped: wait_for_clean 2024-04-25T08:35:47.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:35:47.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:35:47.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:35:47.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:35:47.966 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:35:47.966 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:35:47.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:35:47.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:35:47.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:35:48.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:35:48.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:35:48.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:35:48.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:35:48.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:35:48.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:35:48.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:35:48.621 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:35:48.621 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:35:48.621 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T08:35:48.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:35:48.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:35:48.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:35:48.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836510 2024-04-25T08:35:48.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836510 2024-04-25T08:35:48.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510' 2024-04-25T08:35:48.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:35:48.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:35:48.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-04-25T08:35:48.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-04-25T08:35:48.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672985' 2024-04-25T08:35:48.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:35:48.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:35:49.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509466 2024-04-25T08:35:49.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509466 2024-04-25T08:35:49.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672985 2-64424509466' 2024-04-25T08:35:49.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:35:49.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:35:49.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345927 2024-04-25T08:35:49.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345927 2024-04-25T08:35:49.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672985 2-64424509466 3-85899345927' 2024-04-25T08:35:49.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:35:49.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836510 2024-04-25T08:35:49.228 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:35:49.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:35:49.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836510 2024-04-25T08:35:49.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:35:49.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836510 2024-04-25T08:35:49.230 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836510 2024-04-25T08:35:49.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-04-25T08:35:49.231 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:35:49.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836510 2024-04-25T08:35:49.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:35:49.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-04-25T08:35:49.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:35:49.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:35:49.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-04-25T08:35:49.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:35:49.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-04-25T08:35:49.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-25T08:35:49.634 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672985 2024-04-25T08:35:49.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:35:50.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672983 -lt 42949672985 2024-04-25T08:35:50.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:35:51.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:35:51.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:35:51.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672986 -lt 42949672985 2024-04-25T08:35:51.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:35:51.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509466 2024-04-25T08:35:51.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:35:51.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:35:51.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509466 2024-04-25T08:35:51.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:35:51.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509466 2024-04-25T08:35:51.506 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509466 2024-04-25T08:35:51.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-04-25T08:35:51.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:35:51.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509467 -lt 64424509466 2024-04-25T08:35:51.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:35:51.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345927 2024-04-25T08:35:51.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:35:51.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:35:51.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345927 2024-04-25T08:35:51.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:35:51.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345927 2024-04-25T08:35:51.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345927' 2024-04-25T08:35:51.927 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345927 2024-04-25T08:35:51.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:35:52.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345928 -lt 85899345927 2024-04-25T08:35:52.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:35:52.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:35:52.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:35:52.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:35:52.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:35:52.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:35:52.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:35:52.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:35:52.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:35:52.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:35:52.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:35:53.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:35:53.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:35:53.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:35:53.367 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:35:53.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:35:53.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:35:53.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:35:53.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: seq 1 200 2024-04-25T08:35:53.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:53.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj1 /dev/null 2024-04-25T08:35:54.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:54.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj2 /dev/null 2024-04-25T08:35:54.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:54.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj3 /dev/null 2024-04-25T08:35:54.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:54.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj4 /dev/null 2024-04-25T08:35:54.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:54.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj5 /dev/null 2024-04-25T08:35:55.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:55.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj6 /dev/null 2024-04-25T08:35:55.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:55.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj7 /dev/null 2024-04-25T08:35:55.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:55.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj8 /dev/null 2024-04-25T08:35:55.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:55.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj9 /dev/null 2024-04-25T08:35:56.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:56.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj10 /dev/null 2024-04-25T08:35:56.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:56.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj11 /dev/null 2024-04-25T08:35:56.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:56.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj12 /dev/null 2024-04-25T08:35:56.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:56.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj13 /dev/null 2024-04-25T08:35:56.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:56.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj14 /dev/null 2024-04-25T08:35:57.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:57.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj15 /dev/null 2024-04-25T08:35:57.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:57.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj16 /dev/null 2024-04-25T08:35:57.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:57.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj17 /dev/null 2024-04-25T08:35:57.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:57.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj18 /dev/null 2024-04-25T08:35:58.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:58.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj19 /dev/null 2024-04-25T08:35:58.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:58.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj20 /dev/null 2024-04-25T08:35:58.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:58.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj21 /dev/null 2024-04-25T08:35:58.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:58.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj22 /dev/null 2024-04-25T08:35:58.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:58.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj23 /dev/null 2024-04-25T08:35:59.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:59.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj24 /dev/null 2024-04-25T08:35:59.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:59.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj25 /dev/null 2024-04-25T08:35:59.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:59.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj26 /dev/null 2024-04-25T08:35:59.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:35:59.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj27 /dev/null 2024-04-25T08:36:00.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:00.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj28 /dev/null 2024-04-25T08:36:00.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:00.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj29 /dev/null 2024-04-25T08:36:00.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:00.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj30 /dev/null 2024-04-25T08:36:00.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:00.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj31 /dev/null 2024-04-25T08:36:01.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:01.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj32 /dev/null 2024-04-25T08:36:01.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:01.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj33 /dev/null 2024-04-25T08:36:01.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:01.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj34 /dev/null 2024-04-25T08:36:01.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:01.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj35 /dev/null 2024-04-25T08:36:02.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:02.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj36 /dev/null 2024-04-25T08:36:02.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:02.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj37 /dev/null 2024-04-25T08:36:02.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:02.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj38 /dev/null 2024-04-25T08:36:02.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:02.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj39 /dev/null 2024-04-25T08:36:03.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:03.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj40 /dev/null 2024-04-25T08:36:03.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:03.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj41 /dev/null 2024-04-25T08:36:03.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:03.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj42 /dev/null 2024-04-25T08:36:03.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:03.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj43 /dev/null 2024-04-25T08:36:04.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:04.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj44 /dev/null 2024-04-25T08:36:04.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:04.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj45 /dev/null 2024-04-25T08:36:04.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:04.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj46 /dev/null 2024-04-25T08:36:04.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:04.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj47 /dev/null 2024-04-25T08:36:04.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:04.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj48 /dev/null 2024-04-25T08:36:05.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:05.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj49 /dev/null 2024-04-25T08:36:05.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:05.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj50 /dev/null 2024-04-25T08:36:05.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:05.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj51 /dev/null 2024-04-25T08:36:05.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:05.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj52 /dev/null 2024-04-25T08:36:06.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:06.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj53 /dev/null 2024-04-25T08:36:06.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:06.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj54 /dev/null 2024-04-25T08:36:06.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:06.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj55 /dev/null 2024-04-25T08:36:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj56 /dev/null 2024-04-25T08:36:07.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:07.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj57 /dev/null 2024-04-25T08:36:07.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:07.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj58 /dev/null 2024-04-25T08:36:07.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:07.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj59 /dev/null 2024-04-25T08:36:07.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:07.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj60 /dev/null 2024-04-25T08:36:08.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:08.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj61 /dev/null 2024-04-25T08:36:08.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:08.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj62 /dev/null 2024-04-25T08:36:08.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:08.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj63 /dev/null 2024-04-25T08:36:08.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:08.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj64 /dev/null 2024-04-25T08:36:08.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:08.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj65 /dev/null 2024-04-25T08:36:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:09.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj66 /dev/null 2024-04-25T08:36:09.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:09.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj67 /dev/null 2024-04-25T08:36:09.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:09.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj68 /dev/null 2024-04-25T08:36:09.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:09.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj69 /dev/null 2024-04-25T08:36:10.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:10.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj70 /dev/null 2024-04-25T08:36:10.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:10.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj71 /dev/null 2024-04-25T08:36:10.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:10.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj72 /dev/null 2024-04-25T08:36:10.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:10.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj73 /dev/null 2024-04-25T08:36:11.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:11.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj74 /dev/null 2024-04-25T08:36:11.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:11.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj75 /dev/null 2024-04-25T08:36:11.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:11.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj76 /dev/null 2024-04-25T08:36:11.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:11.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj77 /dev/null 2024-04-25T08:36:12.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:12.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj78 /dev/null 2024-04-25T08:36:12.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:12.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj79 /dev/null 2024-04-25T08:36:12.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:12.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj80 /dev/null 2024-04-25T08:36:12.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:12.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj81 /dev/null 2024-04-25T08:36:12.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:12.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj82 /dev/null 2024-04-25T08:36:13.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:13.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj83 /dev/null 2024-04-25T08:36:13.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:13.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj84 /dev/null 2024-04-25T08:36:13.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:13.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj85 /dev/null 2024-04-25T08:36:13.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:13.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj86 /dev/null 2024-04-25T08:36:14.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:14.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj87 /dev/null 2024-04-25T08:36:14.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:14.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj88 /dev/null 2024-04-25T08:36:14.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:14.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj89 /dev/null 2024-04-25T08:36:14.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:14.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj90 /dev/null 2024-04-25T08:36:14.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:14.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj91 /dev/null 2024-04-25T08:36:15.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:15.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj92 /dev/null 2024-04-25T08:36:15.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:15.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj93 /dev/null 2024-04-25T08:36:15.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:15.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj94 /dev/null 2024-04-25T08:36:15.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:15.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj95 /dev/null 2024-04-25T08:36:16.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:16.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj96 /dev/null 2024-04-25T08:36:16.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:16.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj97 /dev/null 2024-04-25T08:36:16.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:16.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj98 /dev/null 2024-04-25T08:36:16.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:16.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj99 /dev/null 2024-04-25T08:36:17.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:17.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj100 /dev/null 2024-04-25T08:36:17.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:17.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj101 /dev/null 2024-04-25T08:36:17.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:17.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj102 /dev/null 2024-04-25T08:36:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj103 /dev/null 2024-04-25T08:36:17.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:17.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj104 /dev/null 2024-04-25T08:36:18.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:18.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj105 /dev/null 2024-04-25T08:36:18.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:18.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj106 /dev/null 2024-04-25T08:36:18.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:18.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj107 /dev/null 2024-04-25T08:36:18.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:18.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj108 /dev/null 2024-04-25T08:36:19.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:19.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj109 /dev/null 2024-04-25T08:36:19.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:19.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj110 /dev/null 2024-04-25T08:36:19.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:19.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj111 /dev/null 2024-04-25T08:36:19.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:19.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj112 /dev/null 2024-04-25T08:36:20.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:20.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj113 /dev/null 2024-04-25T08:36:20.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:20.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj114 /dev/null 2024-04-25T08:36:20.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:20.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj115 /dev/null 2024-04-25T08:36:20.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:20.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj116 /dev/null 2024-04-25T08:36:21.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:21.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj117 /dev/null 2024-04-25T08:36:21.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:21.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj118 /dev/null 2024-04-25T08:36:21.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:21.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj119 /dev/null 2024-04-25T08:36:21.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:21.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj120 /dev/null 2024-04-25T08:36:21.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:21.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj121 /dev/null 2024-04-25T08:36:22.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:22.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj122 /dev/null 2024-04-25T08:36:22.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:22.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj123 /dev/null 2024-04-25T08:36:22.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:22.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj124 /dev/null 2024-04-25T08:36:22.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:22.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj125 /dev/null 2024-04-25T08:36:23.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:23.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj126 /dev/null 2024-04-25T08:36:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:23.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj127 /dev/null 2024-04-25T08:36:23.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:23.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj128 /dev/null 2024-04-25T08:36:23.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:23.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj129 /dev/null 2024-04-25T08:36:24.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:24.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj130 /dev/null 2024-04-25T08:36:24.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:24.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj131 /dev/null 2024-04-25T08:36:24.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:24.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj132 /dev/null 2024-04-25T08:36:24.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:24.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj133 /dev/null 2024-04-25T08:36:25.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:25.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj134 /dev/null 2024-04-25T08:36:25.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:25.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj135 /dev/null 2024-04-25T08:36:25.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:25.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj136 /dev/null 2024-04-25T08:36:25.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:25.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj137 /dev/null 2024-04-25T08:36:25.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:25.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj138 /dev/null 2024-04-25T08:36:26.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:26.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj139 /dev/null 2024-04-25T08:36:26.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:26.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj140 /dev/null 2024-04-25T08:36:26.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:26.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj141 /dev/null 2024-04-25T08:36:26.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:26.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj142 /dev/null 2024-04-25T08:36:27.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:27.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj143 /dev/null 2024-04-25T08:36:27.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:27.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj144 /dev/null 2024-04-25T08:36:27.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:27.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj145 /dev/null 2024-04-25T08:36:27.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:27.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj146 /dev/null 2024-04-25T08:36:28.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:28.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj147 /dev/null 2024-04-25T08:36:28.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:28.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj148 /dev/null 2024-04-25T08:36:28.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:28.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj149 /dev/null 2024-04-25T08:36:28.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:28.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj150 /dev/null 2024-04-25T08:36:28.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:28.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj151 /dev/null 2024-04-25T08:36:29.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:29.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj152 /dev/null 2024-04-25T08:36:29.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:29.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj153 /dev/null 2024-04-25T08:36:29.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:29.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj154 /dev/null 2024-04-25T08:36:29.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:29.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj155 /dev/null 2024-04-25T08:36:30.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:30.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj156 /dev/null 2024-04-25T08:36:30.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:30.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj157 /dev/null 2024-04-25T08:36:30.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:30.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj158 /dev/null 2024-04-25T08:36:30.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:30.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj159 /dev/null 2024-04-25T08:36:30.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:30.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj160 /dev/null 2024-04-25T08:36:31.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:31.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj161 /dev/null 2024-04-25T08:36:31.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:31.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj162 /dev/null 2024-04-25T08:36:32.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:32.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj163 /dev/null 2024-04-25T08:36:32.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:32.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj164 /dev/null 2024-04-25T08:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj165 /dev/null 2024-04-25T08:36:32.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:32.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj166 /dev/null 2024-04-25T08:36:33.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:33.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj167 /dev/null 2024-04-25T08:36:33.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:33.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj168 /dev/null 2024-04-25T08:36:33.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:33.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj169 /dev/null 2024-04-25T08:36:33.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:33.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj170 /dev/null 2024-04-25T08:36:34.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:34.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj171 /dev/null 2024-04-25T08:36:34.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:34.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj172 /dev/null 2024-04-25T08:36:34.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:34.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj173 /dev/null 2024-04-25T08:36:34.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:34.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj174 /dev/null 2024-04-25T08:36:34.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:34.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj175 /dev/null 2024-04-25T08:36:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:35.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj176 /dev/null 2024-04-25T08:36:35.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:35.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj177 /dev/null 2024-04-25T08:36:35.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:35.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj178 /dev/null 2024-04-25T08:36:35.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:35.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj179 /dev/null 2024-04-25T08:36:36.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:36.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj180 /dev/null 2024-04-25T08:36:36.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:36.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj181 /dev/null 2024-04-25T08:36:36.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:36.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj182 /dev/null 2024-04-25T08:36:36.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:36.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj183 /dev/null 2024-04-25T08:36:37.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:37.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj184 /dev/null 2024-04-25T08:36:37.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:37.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj185 /dev/null 2024-04-25T08:36:37.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:37.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj186 /dev/null 2024-04-25T08:36:37.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:37.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj187 /dev/null 2024-04-25T08:36:38.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:38.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj188 /dev/null 2024-04-25T08:36:38.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:38.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj189 /dev/null 2024-04-25T08:36:38.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:38.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj190 /dev/null 2024-04-25T08:36:38.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:38.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj191 /dev/null 2024-04-25T08:36:39.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:39.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj192 /dev/null 2024-04-25T08:36:39.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:39.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj193 /dev/null 2024-04-25T08:36:39.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:39.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj194 /dev/null 2024-04-25T08:36:39.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:39.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj195 /dev/null 2024-04-25T08:36:39.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:39.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj196 /dev/null 2024-04-25T08:36:40.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:40.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj197 /dev/null 2024-04-25T08:36:40.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:40.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj198 /dev/null 2024-04-25T08:36:40.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj199 /dev/null 2024-04-25T08:36:40.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:36:40.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj200 /dev/null 2024-04-25T08:36:41.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: get_primary test obj1 2024-04-25T08:36:41.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:36:41.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:36:41.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:36:41.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:36:41.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: local primary=1 2024-04-25T08:36:41.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: get_pg test obj1 2024-04-25T08:36:41.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:36:41.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:36:41.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:36:41.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:36:42.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: local PG=1.0 2024-04-25T08:36:42.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: get_not_primary test obj1 2024-04-25T08:36:42.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:36:42.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:36:42.026 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:36:42.026 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:36:42.026 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:36:42.026 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:36:42.027 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:36:42.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:36:42.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:36:42.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:36:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: local otherosd=0 2024-04-25T08:36:42.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:408: TEST_recovery_erasure_remapped: ceph osd set norecover 2024-04-25T08:36:43.532 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is set 2024-04-25T08:36:43.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:36:43.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: kill 295375 2024-04-25T08:36:43.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:410: TEST_recovery_erasure_remapped: ceph osd down osd.0 2024-04-25T08:36:44.088 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T08:36:44.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:411: TEST_recovery_erasure_remapped: ceph osd out osd.0 2024-04-25T08:36:44.973 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. 2024-04-25T08:36:44.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:36:44.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 0 = 1 ']' 2024-04-25T08:36:44.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 0 = 0 ']' 2024-04-25T08:36:44.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:422: TEST_recovery_erasure_remapped: continue 2024-04-25T08:36:44.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:36:44.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 1 = 1 ']' 2024-04-25T08:36:44.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:418: TEST_recovery_erasure_remapped: continue 2024-04-25T08:36:44.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:36:44.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 2 = 1 ']' 2024-04-25T08:36:44.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 2 = 0 ']' 2024-04-25T08:36:44.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:424: TEST_recovery_erasure_remapped: ceph osd out osd.2 2024-04-25T08:36:46.250 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already out. 2024-04-25T08:36:46.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:425: TEST_recovery_erasure_remapped: break 2024-04-25T08:36:46.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:427: TEST_recovery_erasure_remapped: ceph osd unset norecover 2024-04-25T08:36:47.394 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is unset 2024-04-25T08:36:47.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: get_primary test obj1 2024-04-25T08:36:47.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:36:47.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:36:47.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:36:47.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:36:47.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:36:48.028 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:36:48.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:429: TEST_recovery_erasure_remapped: sleep 2 2024-04-25T08:36:50.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:431: TEST_recovery_erasure_remapped: wait_for_clean 2024-04-25T08:36:50.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:36:50.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:36:50.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:36:50.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:36:50.045 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:36:50.045 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:36:50.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:36:50.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:36:50.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:36:50.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:36:50.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:36:50.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:36:50.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:36:50.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:36:50.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:36:50.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:36:50.598 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:36:50.598 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:36:50.599 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T08:36:50.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:36:50.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:50.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:36:50.720 INFO:tasks.workunit.client.0.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:36:50.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:36:50.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:36:50.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:36:50.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:50.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:36:50.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673000 2024-04-25T08:36:50.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673000 2024-04-25T08:36:50.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673000' 2024-04-25T08:36:50.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:50.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:36:51.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509481 2024-04-25T08:36:51.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509481 2024-04-25T08:36:51.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673000 2-64424509481' 2024-04-25T08:36:51.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:51.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:36:51.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345942 2024-04-25T08:36:51.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345942 2024-04-25T08:36:51.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673000 2-64424509481 3-85899345942' 2024-04-25T08:36:51.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:51.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673000 2024-04-25T08:36:51.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:51.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:36:51.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673000 2024-04-25T08:36:51.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:51.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673000 2024-04-25T08:36:51.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2024-04-25T08:36:51.226 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673000 2024-04-25T08:36:51.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:36:51.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672998 -lt 42949673000 2024-04-25T08:36:51.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:36:52.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:36:52.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:36:53.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673000 -lt 42949673000 2024-04-25T08:36:53.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:53.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509481 2024-04-25T08:36:53.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:53.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:36:53.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:53.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509481 2024-04-25T08:36:53.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509481 2024-04-25T08:36:53.175 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509481 2024-04-25T08:36:53.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-04-25T08:36:53.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:36:53.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509481 -lt 64424509481 2024-04-25T08:36:53.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:53.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345942 2024-04-25T08:36:53.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:53.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:36:53.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345942 2024-04-25T08:36:53.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:53.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345942 2024-04-25T08:36:53.683 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345942 2024-04-25T08:36:53.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2024-04-25T08:36:53.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:36:54.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345942 -lt 85899345942 2024-04-25T08:36:54.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:36:54.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:36:54.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:36:54.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:36:54.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:36:54.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:36:54.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:36:54.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:36:54.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:36:54.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:36:54.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:36:55.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:36:55.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:36:55.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:36:55.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:36:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:36:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:36:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:36:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:36:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:36:55.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:36:55.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:36:55.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:36:55.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:36:55.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:36:55.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:36:55.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:36:55.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:36:55.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:36:56.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:36:56.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:36:56.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:36:56.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:36:56.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:36:56.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:36:56.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:36:56.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:36:56.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:36:56.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:36:56.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:36:56.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:36:56.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:36:57.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:36:57.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:36:57.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:36:57.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:36:57.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:36:57.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:36:57.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:36:57.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:36:57.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:36:57.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:36:57.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:36:57.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:36:57.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:36:58.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:36:58.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:36:58.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:36:58.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:36:58.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:36:58.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:36:58.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:36:58.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:36:58.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:36:58.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:36:58.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:36:58.640 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:36:58.640 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:36:59.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:36:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:36:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:36:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:36:59.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:36:59.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:36:59.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:36:59.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:36:59.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:36:59.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:36:59.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:36:59.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:36:59.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:36:59.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:36:59.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:36:59.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:36:59.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:00.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:00.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:00.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:00.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:00.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:00.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:00.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:00.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:00.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:01.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:37:01.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:37:01.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:01.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:01.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:01.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:01.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:01.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:01.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:01.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:01.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:01.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:01.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:01.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:01.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:01.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:01.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:02.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:02.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:02.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:02.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:02.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:02.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:02.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:02.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:02.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:02.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13 2024-04-25T08:37:02.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13 '!=' null 2024-04-25T08:37:02.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:02.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:02.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:03.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:03.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:03.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:03.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:03.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:03.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:03.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:03.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:03.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:37:03.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:03.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:03.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:04.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:37:04.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:37:04.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:37:04.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:37:04.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:37:04.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:37:04.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:37:04.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:37:04.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:37:04.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-04-25T08:37:04.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-04-25T08:37:04.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:37:04.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:37:04.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:37:04.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:37:04.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:37:04.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:37:04.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:37:04.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:37:04.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:37:04.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:37:04.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:37:05.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:37:05.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:37:05.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:37:05.219 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:37:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:37:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:37:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:37:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:433: TEST_recovery_erasure_remapped: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:37:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:434: TEST_recovery_erasure_remapped: check td/osd-recovery-stats 1.0 1 erasure 200 0 200 200 2024-04-25T08:37:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=200 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=200 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:37:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:37:05.762 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:37:05.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,NONE]/[1,NONE,2]' 2024-04-25T08:37:05.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:37:05.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:37:05.777 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:37:05.777 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-04-25T08:37:05.777 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:37:05.777 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:37:05.777 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:37:05.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:37:05.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:37:05.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:37:05.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:37:05.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:37:05.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:37:05.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:37:05.787 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:37:05.787 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:37:05.787 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:37:05.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:37:05.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:37:05.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:37:05.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:37:05.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:37:05.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:37:05.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:37:05.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:37:05.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:37:05.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-04-25T08:37:05.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:37:05.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:37:05.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:37:05.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=200 2024-04-25T08:37:05.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 200 200 2024-04-25T08:37:05.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:37:05.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:37:05.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:37:05.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:37:05.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:37:05.810 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:37:05.810 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:37:05.810 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:37:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=200 2024-04-25T08:37:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 200 200 2024-04-25T08:37:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=200 2024-04-25T08:37:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:37:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=200 2024-04-25T08:37:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:37:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:37:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:436: TEST_recovery_erasure_remapped: delete_pool test 2024-04-25T08:37:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:37:05.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:37:06.321 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T08:37:06.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:437: TEST_recovery_erasure_remapped: kill_daemons td/osd-recovery-stats 2024-04-25T08:37:06.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:37:06.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:37:06.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:37:06.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:37:06.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:37:11.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:37:11.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:37:11.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:37:11.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:37:11.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:37:11.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:37:11.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:37:11.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:37:11.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:37:11.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:37:11.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:37:11.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:37:11.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:37:11.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:37:11.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:37:11.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:37:11.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:37:11.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:37:11.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:37:11.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:37:11.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:37:11.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:37:11.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:37:11.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:37:11.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:37:11.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:11.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:37:11.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:37:11.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:37:11.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:37:11.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:37:11.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:37:11.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:37:11.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:37:11.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:37:11.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:37:11.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:37:11.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:37:11.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:37:11.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:37:11.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:37:11.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:37:11.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:37:11.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:37:11.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:37:11.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:37:11.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:37:11.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:37:11.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:37:11.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:37:11.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:37:11.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:37:11.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:37:11.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:37:11.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:37:11.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:37:11.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:37:11.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:11.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:37:11.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:37:11.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:37:11.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:37:11.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:37:11.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:37:11.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:11.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:37:11.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.277516 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_multi td/osd-recovery-stats 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:441: TEST_recovery_multi: local dir=td/osd-recovery-stats 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:443: TEST_recovery_multi: local osds=6 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:444: TEST_recovery_multi: run_mon td/osd-recovery-stats a 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:37:11.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:37:11.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:37:11.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:37:12.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:37:12.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:37:12.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:37:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:37:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:37:12.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:37:12.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:37:12.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:37:12.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:37:12.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:37:12.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:37:12.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:37:12.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:37:12.513 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:37:12.513 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:37:12.513 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:37:12.513 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:37:12.513 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:12.513 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:37:12.513 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:37:12.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:37:12.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get fsid 2024-04-25T08:37:12.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:37:12.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:37:12.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:37:12.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:37:12.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:37:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:37:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:37:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:37:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:37:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:12.629 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:37:12.630 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:37:12.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:37:12.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get mon_host 2024-04-25T08:37:12.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:445: TEST_recovery_multi: run_mgr td/osd-recovery-stats x 2024-04-25T08:37:12.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:37:12.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:37:12.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:37:12.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:37:12.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:37:12.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:37:13.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:37:13.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:37:13.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:37:13.650 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:37:13.650 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:13.650 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:37:13.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:37:13.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:37:13.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:37:13.686 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: expr 6 - 1 2024-04-25T08:37:13.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: seq 0 5 2024-04-25T08:37:13.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 0 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:37:13.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:37:13.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:37:13.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:37:13.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:37:13.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:37:13.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:37:13.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:37:13.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:37:13.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:37:13.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:37:13.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=38dbd4be-4ad1-4d36-ab43-b04d4b907711 2024-04-25T08:37:13.694 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 38dbd4be-4ad1-4d36-ab43-b04d4b907711 2024-04-25T08:37:13.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 38dbd4be-4ad1-4d36-ab43-b04d4b907711' 2024-04-25T08:37:13.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:37:13.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5FipmvOC+KhAAsE43wOGFpLnLCisA1ECrNg== 2024-04-25T08:37:13.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5FipmvOC+KhAAsE43wOGFpLnLCisA1ECrNg=="}' 2024-04-25T08:37:13.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 38dbd4be-4ad1-4d36-ab43-b04d4b907711 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:37:14.022 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:37:14.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:37:14.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5FipmvOC+KhAAsE43wOGFpLnLCisA1ECrNg== --osd-uuid 38dbd4be-4ad1-4d36-ab43-b04d4b907711 2024-04-25T08:37:14.072 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:14.077+0000 7fac608817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:14.108 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:14.113+0000 7fac608817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:14.109 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:14.113+0000 7fac608817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:14.110 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:14.113+0000 7fac608817c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:37:14.110 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:14.113+0000 7fac608817c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:37:14.111 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:14.113+0000 7fac608817c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:37:14.156 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:14.161+0000 7fac608817c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:37:20.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:37:20.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:37:20.654 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:37:20.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:37:20.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:37:21.215 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:37:21.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:37:21.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:37:21.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:37:21.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:37:21.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:37:21.246 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:21.249+0000 7fa5b09757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:21.281 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:21.285+0000 7fa5b09757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:21.283 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:21.289+0000 7fa5b09757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:21.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:37:21.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:37:21.637 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:37:21.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:37:21.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:37:21.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:37:21.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:21.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:37:21.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:21.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:21.857 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:21.861+0000 7fa5b09757c0 -1 Falling back to public interface 2024-04-25T08:37:22.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:23.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:23.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:23.048 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:37:23.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:37:23.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:23.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:23.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:24.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:24.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:24.454 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:37:24.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:37:24.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:24.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:24.488 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:24.493+0000 7fa5b09757c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:37:24.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:25.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:25.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:25.992 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:37:25.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:37:25.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:25.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:26.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:27.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:27.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:27.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:37:27.468 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:37:27.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:27.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:27.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:28.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:28.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:28.926 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:37:28.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:37:28.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:28.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:29.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:30.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:30.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:30.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:37:30.475 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:37:30.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:30.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:30.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:31.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:31.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:31.949 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:37:31.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:37:31.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:31.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:32.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:33.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:33.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:33.543 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:37:33.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:37:33.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:33.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:33.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:34.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:34.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:34.984 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:37:34.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:37:34.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:34.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:35.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:36.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:36.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:36.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:37:36.467 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:37:36.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:36.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:36.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:37.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:37.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:37.929 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:37:37.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:37:37.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:37.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:38.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:39.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:39.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:39.909 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:37:39.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:37:39.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:39.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:40.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:41.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:41.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:41.666 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:37:41.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:37:41.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:41.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:42.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:43.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:43.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:43.432 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:37:43.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:37:43.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:43.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:45.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:45.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:45.077 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:37:45.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:37:45.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:45.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:45.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:46.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:46.997 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:37:46.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:37:46.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:46.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:47.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:48.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:48.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:48.867 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:37:48.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:37:48.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:48.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:49.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:50.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:50.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:50.332 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:37:50.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:37:50.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:50.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:50.829 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2786072826,v1:127.0.0.1:6803/2786072826] [v2:127.0.0.1:6804/2786072826,v1:127.0.0.1:6805/2786072826] exists,up 38dbd4be-4ad1-4d36-ab43-b04d4b907711 2024-04-25T08:37:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:37:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 1 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:37:50.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:37:50.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:37:50.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:37:50.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:37:50.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:37:50.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:37:50.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:37:50.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:37:50.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:37:50.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:37:50.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:37:50.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2c7386c4-296c-4b50-afff-2a1635217bfa 2024-04-25T08:37:50.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2c7386c4-296c-4b50-afff-2a1635217bfa' 2024-04-25T08:37:50.835 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 2c7386c4-296c-4b50-afff-2a1635217bfa 2024-04-25T08:37:50.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:37:50.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeFipmbsx9MxAAu6zeaSZn+BNKMelLs1jziQ== 2024-04-25T08:37:50.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeFipmbsx9MxAAu6zeaSZn+BNKMelLs1jziQ=="}' 2024-04-25T08:37:50.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2c7386c4-296c-4b50-afff-2a1635217bfa -i td/osd-recovery-stats/1/new.json 2024-04-25T08:37:51.422 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:37:51.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:37:51.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBeFipmbsx9MxAAu6zeaSZn+BNKMelLs1jziQ== --osd-uuid 2c7386c4-296c-4b50-afff-2a1635217bfa 2024-04-25T08:37:51.466 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:51.468+0000 7f8d7812c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:51.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:51.512+0000 7f8d7812c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:51.509 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:51.512+0000 7f8d7812c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:51.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:51.512+0000 7f8d7812c7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:37:51.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:51.512+0000 7f8d7812c7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:37:51.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:51.512+0000 7f8d7812c7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:37:51.547 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:51.552+0000 7f8d7812c7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:37:58.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:37:58.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:37:58.477 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T08:37:58.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:37:58.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:37:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:37:59.049 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:37:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:37:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:37:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:37:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:37:59.076 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:59.080+0000 7f9d41c927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:59.078 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:59.080+0000 7f9d41c927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:59.079 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:59.084+0000 7f9d41c927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:59.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:37:59.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:37:59.470 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:37:59.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:37:59.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:37:59.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:37:59.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:59.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:37:59.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:59.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:59.632 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:37:59.636+0000 7f9d41c927c0 -1 Falling back to public interface 2024-04-25T08:37:59.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:00.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:00.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:00.895 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:38:00.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:38:00.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:00.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:01.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:02.251 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:02.256+0000 7f9d41c927c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:38:02.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:02.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:02.298 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:38:02.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:38:02.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:02.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:02.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:03.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:03.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:03.859 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:38:03.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:38:03.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:03.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:04.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:05.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:05.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:05.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:38:05.339 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:38:05.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:05.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:05.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:06.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:06.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:06.776 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:38:06.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:38:06.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:06.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:07.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:08.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:08.183 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:38:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:38:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:08.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:08.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:09.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:09.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:09.754 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:38:09.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:38:09.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:09.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:10.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:11.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:11.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:11.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:38:11.338 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:38:11.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:11.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:11.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:12.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:12.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:12.825 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:38:12.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:38:12.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:12.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:13.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:14.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:14.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:14.491 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:38:14.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:38:14.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:14.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:15.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:16.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:16.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:16.086 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:38:16.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:38:16.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:16.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:16.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:17.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:17.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:17.507 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:38:17.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:38:17.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:17.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:18.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:19.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:19.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:19.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:38:19.407 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:38:19.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:19.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:20.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:21.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:21.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:21.199 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:38:21.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:38:21.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:21.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:21.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:22.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:22.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:22.899 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:38:22.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:38:22.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:22.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:23.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:24.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:24.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:24.799 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:38:24.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:38:24.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:24.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:25.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:26.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:26.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:26.560 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:38:26.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:38:26.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:26.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:27.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:28.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:28.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:28.551 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:38:28.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:38:28.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:28.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:38:29.089 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/385728416,v1:127.0.0.1:6811/385728416] [v2:127.0.0.1:6812/385728416,v1:127.0.0.1:6813/385728416] exists,up 2c7386c4-296c-4b50-afff-2a1635217bfa 2024-04-25T08:38:29.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:38:29.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 2 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:38:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:38:29.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:38:29.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:38:29.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:38:29.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:38:29.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:38:29.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:38:29.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:38:29.093 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:38:29.093 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:29.093 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:38:29.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:38:29.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:38:29.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:38:29.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:38:29.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:38:29.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:38:29.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:38:29.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:38:29.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:38:29.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:38:29.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:38:29.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:38:29.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:38:29.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:38:29.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:38:29.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eb41c6df-0872-4528-b6d4-846bef778199 2024-04-25T08:38:29.096 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 eb41c6df-0872-4528-b6d4-846bef778199 2024-04-25T08:38:29.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 eb41c6df-0872-4528-b6d4-846bef778199' 2024-04-25T08:38:29.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:38:29.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFFipmulCTBxAAVu24/k8uyciaIl9K8rxYcQ== 2024-04-25T08:38:29.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFFipmulCTBxAAVu24/k8uyciaIl9K8rxYcQ=="}' 2024-04-25T08:38:29.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eb41c6df-0872-4528-b6d4-846bef778199 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:38:29.679 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:38:29.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:38:29.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFFipmulCTBxAAVu24/k8uyciaIl9K8rxYcQ== --osd-uuid eb41c6df-0872-4528-b6d4-846bef778199 2024-04-25T08:38:29.729 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:29.735+0000 7ff02b7b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:29.764 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:29.771+0000 7ff02b7b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:29.765 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:29.771+0000 7ff02b7b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:29.766 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:29.771+0000 7ff02b7b27c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:38:29.766 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:29.771+0000 7ff02b7b27c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:38:29.766 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:29.771+0000 7ff02b7b27c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:38:29.803 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:29.811+0000 7ff02b7b27c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:38:36.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:38:36.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:38:36.710 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T08:38:36.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:38:36.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:38:37.334 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T08:38:37.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:38:37.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:38:37.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:38:37.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:38:37.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:38:37.368 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:37.375+0000 7f3f7da177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:37.412 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:37.419+0000 7f3f7da177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:37.413 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:37.419+0000 7f3f7da177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:37.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:38:37.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:38:37.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:38:37.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:38:37.732 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:38:37.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:38:37.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:37.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:38:37.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:37.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:37.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:37.979+0000 7f3f7da177c0 -1 Falling back to public interface 2024-04-25T08:38:38.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:39.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:39.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:39.245 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:38:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:38:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:39.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:40.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:40.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:40.645 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:38:40.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:38:40.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:40.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:40.729 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:38:40.735+0000 7f3f7da177c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:38:41.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:38:42.122 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:38:42.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:42.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:42.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:43.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:43.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:43.735 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:38:43.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:38:43.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:43.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:44.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:45.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:45.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:45.191 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:38:45.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:38:45.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:45.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:45.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:46.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:46.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:46.697 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:38:46.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:38:46.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:46.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:47.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:48.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:48.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:48.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:38:48.299 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:38:48.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:48.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:48.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:49.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:49.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:49.848 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:38:49.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:38:49.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:49.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:50.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:51.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:51.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:51.314 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:38:51.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:38:51.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:51.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:51.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:52.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:52.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:52.909 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:38:52.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:38:52.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:52.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:53.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:54.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:54.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:54.450 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:38:54.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:38:54.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:54.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:55.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:56.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:56.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:56.070 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:38:56.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:38:56.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:56.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:56.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:57.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:57.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:57.623 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:38:57.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:38:57.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:57.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:58.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:59.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:59.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:59.782 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:38:59.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:38:59.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:59.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:00.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:01.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:01.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:01.600 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:39:01.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:39:01.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:01.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:02.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:03.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:03.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:03.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:39:03.462 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:39:03.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:03.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:04.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:05.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:05.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:05.374 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:39:05.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:39:05.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:05.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:06.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:07.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:07.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:07.246 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:39:07.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:39:07.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:07.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:07.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:08.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:08.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:08.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:39:08.755 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:39:08.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:08.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:39:09.199 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/853707759,v1:127.0.0.1:6819/853707759] [v2:127.0.0.1:6820/853707759,v1:127.0.0.1:6821/853707759] exists,up eb41c6df-0872-4528-b6d4-846bef778199 2024-04-25T08:39:09.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:39:09.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:39:09.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:39:09.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 3 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:39:09.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:39:09.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:39:09.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:39:09.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:39:09.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:39:09.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:39:09.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:39:09.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:39:09.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:39:09.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=70f49c13-901b-42a7-bdb2-e6ebedd1f6b6 2024-04-25T08:39:09.206 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 70f49c13-901b-42a7-bdb2-e6ebedd1f6b6 2024-04-25T08:39:09.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 70f49c13-901b-42a7-bdb2-e6ebedd1f6b6' 2024-04-25T08:39:09.206 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:39:09.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtFipmhdbuDRAAoj7BbdcvGS0qnfgffUhlYg== 2024-04-25T08:39:09.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtFipmhdbuDRAAoj7BbdcvGS0qnfgffUhlYg=="}' 2024-04-25T08:39:09.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 70f49c13-901b-42a7-bdb2-e6ebedd1f6b6 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:39:09.857 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:39:09.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:39:09.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtFipmhdbuDRAAoj7BbdcvGS0qnfgffUhlYg== --osd-uuid 70f49c13-901b-42a7-bdb2-e6ebedd1f6b6 2024-04-25T08:39:09.913 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:09.919+0000 7efcc7afc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:09.973 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:09.979+0000 7efcc7afc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:09.975 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:09.983+0000 7efcc7afc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:09.975 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:09.983+0000 7efcc7afc7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:39:09.975 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:09.983+0000 7efcc7afc7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:39:09.976 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:09.983+0000 7efcc7afc7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:39:10.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:10.043+0000 7efcc7afc7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:39:16.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:39:16.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:39:16.724 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T08:39:16.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:39:16.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:39:17.270 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T08:39:17.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:39:17.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:39:17.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:39:17.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:39:17.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:39:17.301 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:17.306+0000 7fe2e52497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:17.303 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:17.310+0000 7fe2e52497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:17.304 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:17.310+0000 7fe2e52497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:17.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:39:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:39:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:39:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:39:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:39:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:39:17.721 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:39:17.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:17.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:17.879 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:17.886+0000 7fe2e52497c0 -1 Falling back to public interface 2024-04-25T08:39:18.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:19.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:19.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:19.177 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:39:19.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:39:19.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:19.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:19.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:20.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:20.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:20.577 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:39:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:39:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:20.601 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:20.606+0000 7fe2e52497c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:39:21.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:22.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:22.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:22.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:39:22.004 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:39:22.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:22.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:22.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:23.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:23.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:23.624 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:39:23.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:39:23.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:23.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:24.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:25.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:25.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:25.160 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:39:25.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:39:25.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:25.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:25.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:26.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:26.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:26.653 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:39:26.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:39:26.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:26.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:27.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:28.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:28.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:28.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:39:28.256 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:39:28.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:28.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:28.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:29.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:29.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:29.917 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:39:29.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:39:29.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:29.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:30.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:31.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:31.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:31.327 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:39:31.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:39:31.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:31.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:31.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:32.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:32.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:32.933 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:39:32.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:39:32.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:32.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:33.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:34.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:34.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:34.349 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:39:34.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:39:34.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:34.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:35.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:36.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:36.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:36.116 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:39:36.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:39:36.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:36.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:37.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:38.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:38.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:38.068 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:39:38.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:39:38.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:38.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:39.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:40.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:40.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:40.237 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:39:40.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:39:40.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:40.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:40.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:41.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:41.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:41.649 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:39:41.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:39:41.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:41.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:42.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:43.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:43.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:43.515 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:39:43.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:39:43.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:43.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:44.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:45.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:45.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:45.259 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:39:45.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:39:45.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:45.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:45.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:46.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:46.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:46.848 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:39:46.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:39:46.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:46.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:47.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:48.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:48.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:48.496 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:39:48.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:39:48.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:48.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:48.940 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3436425981,v1:127.0.0.1:6827/3436425981] [v2:127.0.0.1:6828/3436425981,v1:127.0.0.1:6829/3436425981] exists,up 70f49c13-901b-42a7-bdb2-e6ebedd1f6b6 2024-04-25T08:39:48.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:39:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:39:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:39:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:39:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 4 2024-04-25T08:39:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:39:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:39:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:39:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:39:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:39:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:39:48.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:39:48.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:39:48.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:39:48.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:39:48.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:39:48.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:39:48.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:39:48.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:39:48.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:39:48.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:39:48.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:39:48.946 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:39:48.947 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:39:48.947 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:39:48.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:39:48.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:39:48.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:39:48.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:39:48.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:39:48.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:39:48.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:39:48.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:39:48.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:39:48.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:39:48.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:39:48.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:39:48.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:39:48.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:39:48.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:39:48.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dab9e50c-1851-4913-8cb1-096f450e61c0 2024-04-25T08:39:48.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 dab9e50c-1851-4913-8cb1-096f450e61c0' 2024-04-25T08:39:48.949 INFO:tasks.workunit.client.0.smithi102.stdout:add osd4 dab9e50c-1851-4913-8cb1-096f450e61c0 2024-04-25T08:39:48.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:39:48.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUFipm74hWOhAAgSYw7neVNrDiO3Tg4wTitQ== 2024-04-25T08:39:48.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUFipm74hWOhAAgSYw7neVNrDiO3Tg4wTitQ=="}' 2024-04-25T08:39:48.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dab9e50c-1851-4913-8cb1-096f450e61c0 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:39:49.489 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:39:49.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:39:49.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUFipm74hWOhAAgSYw7neVNrDiO3Tg4wTitQ== --osd-uuid dab9e50c-1851-4913-8cb1-096f450e61c0 2024-04-25T08:39:49.542 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:49.550+0000 7fe80395d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:49.587 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:49.594+0000 7fe80395d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:49.589 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:49.594+0000 7fe80395d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:49.589 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:49.594+0000 7fe80395d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:39:49.590 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:49.594+0000 7fe80395d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:39:49.590 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:49.594+0000 7fe80395d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:39:49.651 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:49.658+0000 7fe80395d7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:39:56.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:39:56.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:39:56.709 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd4 key to auth repository 2024-04-25T08:39:56.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:39:56.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:39:57.298 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.4 2024-04-25T08:39:57.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:39:57.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:39:57.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:39:57.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:39:57.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:39:57.331 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:57.338+0000 7fd8ed8f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:57.337 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:57.342+0000 7fd8ed8f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:57.338 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:57.346+0000 7fd8ed8f27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:39:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:39:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:39:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:39:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:39:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:39:57.727 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:39:57.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:57.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:57.887 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:39:57.894+0000 7fd8ed8f27c0 -1 Falling back to public interface 2024-04-25T08:39:58.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:59.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:59.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:59.164 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:39:59.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:39:59.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:59.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:59.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:00.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:00.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:00.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:40:00.587 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:40:00.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:00.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:00.659 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:00.666+0000 7fd8ed8f27c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:40:01.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:02.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:02.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:02.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:40:02.056 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:40:02.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:02.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:02.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:03.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:03.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:03.518 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:40:03.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:40:03.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:03.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:03.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:04.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:04.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:04.977 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:40:04.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:40:04.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:04.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:05.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:06.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:06.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:06.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:40:06.499 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:40:06.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:06.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:07.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:08.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:08.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:08.298 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:40:08.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:40:08.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:08.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:08.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:09.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:09.882 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:40:09.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:09.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:40:09.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:09.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:10.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:11.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:11.353 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:40:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:40:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:11.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:11.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:12.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:12.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:12.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:40:12.972 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:40:12.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:12.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:13.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:14.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:14.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:14.683 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:40:14.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:40:14.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:14.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:15.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:16.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:16.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:16.227 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:40:16.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:40:16.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:16.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:17.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:18.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:18.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:18.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:40:18.278 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:40:18.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:18.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:19.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:20.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:20.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:20.220 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:40:20.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:40:20.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:20.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:21.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:22.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:22.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:22.140 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:40:22.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:40:22.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:22.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:23.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:24.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:24.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:24.719 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:40:24.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:40:24.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:24.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:25.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:26.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:26.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:26.599 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:40:26.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:40:26.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:26.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:27.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:28.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:28.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:28.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:40:28.108 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:40:28.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:28.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:40:28.703 INFO:tasks.workunit.client.0.smithi102.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2470137186,v1:127.0.0.1:6835/2470137186] [v2:127.0.0.1:6836/2470137186,v1:127.0.0.1:6837/2470137186] exists,up dab9e50c-1851-4913-8cb1-096f450e61c0 2024-04-25T08:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 5 2024-04-25T08:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:40:28.706 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:40:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:40:28.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:40:28.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:40:28.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b506ca8c-8324-4fe0-82a3-7983673b559f 2024-04-25T08:40:28.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 b506ca8c-8324-4fe0-82a3-7983673b559f' 2024-04-25T08:40:28.709 INFO:tasks.workunit.client.0.smithi102.stdout:add osd5 b506ca8c-8324-4fe0-82a3-7983673b559f 2024-04-25T08:40:28.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:40:28.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8FipmxXI8LBAAP38wYvxLi/080efPvEkZtA== 2024-04-25T08:40:28.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8FipmxXI8LBAAP38wYvxLi/080efPvEkZtA=="}' 2024-04-25T08:40:28.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b506ca8c-8324-4fe0-82a3-7983673b559f -i td/osd-recovery-stats/5/new.json 2024-04-25T08:40:29.454 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:40:29.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:40:29.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8FipmxXI8LBAAP38wYvxLi/080efPvEkZtA== --osd-uuid b506ca8c-8324-4fe0-82a3-7983673b559f 2024-04-25T08:40:29.511 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:29.517+0000 7f3bc72037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:29.551 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:29.557+0000 7f3bc72037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:29.553 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:29.557+0000 7f3bc72037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:29.553 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:29.557+0000 7f3bc72037c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:40:29.554 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:29.557+0000 7f3bc72037c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:40:29.554 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:29.557+0000 7f3bc72037c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:40:29.660 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:29.665+0000 7f3bc72037c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:40:36.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:40:36.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:40:36.716 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd5 key to auth repository 2024-04-25T08:40:36.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:40:36.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:40:37.314 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.5 2024-04-25T08:40:37.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:40:37.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:40:37.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:40:37.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:40:37.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:40:37.347 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:37.353+0000 7f74d135e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:37.349 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:37.353+0000 7f74d135e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:37.350 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:37.357+0000 7f74d135e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:40:37.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:40:37.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:40:37.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:40:37.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:40:37.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:40:37.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:37.701 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:40:37.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:40:37.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:37.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:37.911 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:37.917+0000 7f74d135e7c0 -1 Falling back to public interface 2024-04-25T08:40:38.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:39.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:39.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:39.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:40:39.143 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:40:39.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:39.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:39.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:40.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:40.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:40.568 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:40:40.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:40:40.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:40.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:40.637 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:40:40.641+0000 7f74d135e7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:40:41.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:42.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:42.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:42.032 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:40:42.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:40:42.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:42.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:42.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:43.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:43.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:43.523 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:40:43.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:40:43.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:43.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:43.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:44.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:44.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:44.995 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:40:44.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:40:44.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:44.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:45.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:46.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:46.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:46.457 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:40:46.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:40:46.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:46.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:46.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:47.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:40:47.955 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:40:47.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:47.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:48.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:49.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:49.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:49.552 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:40:49.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:40:49.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:49.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:50.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:51.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:51.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:51.036 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:40:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:40:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:51.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:51.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:52.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:52.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:52.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:40:52.587 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:40:52.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:52.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:53.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:54.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:54.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:54.146 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:40:54.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:40:54.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:54.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:54.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:55.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:55.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:55.742 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:40:55.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:40:55.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:55.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:56.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:57.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:57.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:57.245 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:40:57.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:40:57.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:57.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:58.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:59.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:59.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:59.047 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:40:59.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:40:59.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:59.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:59.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:00.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:00.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:00.789 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:41:00.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:41:00.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:00.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:01.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:02.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:02.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:02.681 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:41:02.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:41:02.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:02.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:03.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:04.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:04.563 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:41:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:41:04.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:04.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:05.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:06.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:06.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:06.847 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:41:06.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:41:06.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:06.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:07.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:08.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:08.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:08.697 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:41:08.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:41:08.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:08.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:41:09.107 INFO:tasks.workunit.client.0.smithi102.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/4048909503,v1:127.0.0.1:6843/4048909503] [v2:127.0.0.1:6844/4048909503,v1:127.0.0.1:6845/4048909503] exists,up b506ca8c-8324-4fe0-82a3-7983673b559f 2024-04-25T08:41:09.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:41:09.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:41:09.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:41:09.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:451: TEST_recovery_multi: create_pool test 1 1 2024-04-25T08:41:09.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:41:09.677 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T08:41:09.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:41:10.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:452: TEST_recovery_multi: ceph osd pool set test size 3 2024-04-25T08:41:12.058 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 3 2024-04-25T08:41:12.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:453: TEST_recovery_multi: ceph osd pool set test min_size 1 2024-04-25T08:41:13.314 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 min_size to 1 2024-04-25T08:41:13.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:455: TEST_recovery_multi: wait_for_clean 2024-04-25T08:41:13.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:41:13.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:41:13.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:41:13.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:41:13.337 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:41:13.337 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:41:13.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:41:13.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:41:13.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:41:13.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:41:13.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:41:13.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:41:13.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:41:13.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:41:13.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:41:14.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:41:14.206 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:41:14.206 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:41:14.206 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:41:14.206 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T08:41:14.206 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T08:41:14.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:41:14.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:14.207 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:41:14.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836530 2024-04-25T08:41:14.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836530 2024-04-25T08:41:14.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530' 2024-04-25T08:41:14.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:14.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:41:14.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672999 2024-04-25T08:41:14.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672999 2024-04-25T08:41:14.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949672999' 2024-04-25T08:41:14.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:14.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:41:14.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509477 2024-04-25T08:41:14.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509477 2024-04-25T08:41:14.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949672999 2-64424509477' 2024-04-25T08:41:14.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:14.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:41:14.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345942 2024-04-25T08:41:14.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345942 2024-04-25T08:41:14.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949672999 2-64424509477 3-85899345942' 2024-04-25T08:41:14.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:14.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:41:15.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182419 2024-04-25T08:41:15.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182419 2024-04-25T08:41:15.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949672999 2-64424509477 3-85899345942 4-107374182419' 2024-04-25T08:41:15.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:15.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:41:15.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018890 2024-04-25T08:41:15.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018890 2024-04-25T08:41:15.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949672999 2-64424509477 3-85899345942 4-107374182419 5-128849018890' 2024-04-25T08:41:15.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:41:15.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836530 2024-04-25T08:41:15.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:41:15.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:41:15.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836530 2024-04-25T08:41:15.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:41:15.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836530 2024-04-25T08:41:15.243 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836530 2024-04-25T08:41:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-04-25T08:41:15.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:41:15.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836528 -lt 21474836530 2024-04-25T08:41:15.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:41:16.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:41:16.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:41:17.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836531 -lt 21474836530 2024-04-25T08:41:17.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:41:17.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672999 2024-04-25T08:41:17.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:41:17.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:41:17.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672999 2024-04-25T08:41:17.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:41:17.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672999 2024-04-25T08:41:17.066 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672999 2024-04-25T08:41:17.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2024-04-25T08:41:17.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:41:17.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673000 -lt 42949672999 2024-04-25T08:41:17.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:41:17.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509477 2024-04-25T08:41:17.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:41:17.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:41:17.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509477 2024-04-25T08:41:17.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:41:17.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509477 2024-04-25T08:41:17.509 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509477 2024-04-25T08:41:17.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509477' 2024-04-25T08:41:17.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:41:17.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509477 -lt 64424509477 2024-04-25T08:41:17.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:41:17.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345942 2024-04-25T08:41:17.932 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:41:17.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:41:17.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345942 2024-04-25T08:41:17.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:41:17.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345942 2024-04-25T08:41:17.936 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345942 2024-04-25T08:41:17.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2024-04-25T08:41:17.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:41:18.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345943 -lt 85899345942 2024-04-25T08:41:18.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:41:18.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182419 2024-04-25T08:41:18.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:41:18.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:41:18.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182419 2024-04-25T08:41:18.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:41:18.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182419 2024-04-25T08:41:18.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182419' 2024-04-25T08:41:18.389 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182419 2024-04-25T08:41:18.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:41:18.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182419 -lt 107374182419 2024-04-25T08:41:18.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:41:18.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:41:18.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018890 2024-04-25T08:41:18.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:41:18.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018890 2024-04-25T08:41:18.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:41:18.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018890 2024-04-25T08:41:18.807 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018890 2024-04-25T08:41:18.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018890' 2024-04-25T08:41:18.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:41:19.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018890 -lt 128849018890 2024-04-25T08:41:19.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:41:19.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:41:19.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:41:19.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:41:19.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:41:19.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:41:19.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:41:19.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:41:19.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:41:19.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:41:19.806 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:41:20.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:41:20.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:41:20.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:41:20.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:41:20.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:41:20.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:41:20.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:41:20.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:457: TEST_recovery_multi: rados -p test put obj1 /dev/null 2024-04-25T08:41:20.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:41:20.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:41:20.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:41:20.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:41:20.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:41:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: local primary=1 2024-04-25T08:41:21.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: get_not_primary test obj1 2024-04-25T08:41:21.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:41:21.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:41:21.323 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:41:21.323 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:41:21.323 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:41:21.324 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:41:21.324 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:41:21.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:41:21.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:41:21.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:41:22.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: local otherosd=0 2024-04-25T08:41:22.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:462: TEST_recovery_multi: ceph osd set noout 2024-04-25T08:41:22.979 INFO:tasks.workunit.client.0.smithi102.stderr:noout is set 2024-04-25T08:41:22.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:463: TEST_recovery_multi: ceph osd set norecover 2024-04-25T08:41:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is set 2024-04-25T08:41:23.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:41:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: kill 308347 2024-04-25T08:41:23.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:465: TEST_recovery_multi: ceph osd down osd.0 2024-04-25T08:41:24.403 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T08:41:24.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: expr 200 / 2 2024-04-25T08:41:24.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: local half=100 2024-04-25T08:41:24.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: seq 2 100 2024-04-25T08:41:24.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:24.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj2 /dev/null 2024-04-25T08:41:25.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:25.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj3 /dev/null 2024-04-25T08:41:25.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:25.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj4 /dev/null 2024-04-25T08:41:25.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:25.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj5 /dev/null 2024-04-25T08:41:25.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:25.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj6 /dev/null 2024-04-25T08:41:26.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:26.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj7 /dev/null 2024-04-25T08:41:26.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:26.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj8 /dev/null 2024-04-25T08:41:26.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:26.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj9 /dev/null 2024-04-25T08:41:26.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:26.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj10 /dev/null 2024-04-25T08:41:26.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:26.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj11 /dev/null 2024-04-25T08:41:26.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:26.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj12 /dev/null 2024-04-25T08:41:26.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:26.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj13 /dev/null 2024-04-25T08:41:27.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:27.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj14 /dev/null 2024-04-25T08:41:27.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:27.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj15 /dev/null 2024-04-25T08:41:27.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:27.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj16 /dev/null 2024-04-25T08:41:27.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:27.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj17 /dev/null 2024-04-25T08:41:27.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:27.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj18 /dev/null 2024-04-25T08:41:27.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:27.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj19 /dev/null 2024-04-25T08:41:27.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:27.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj20 /dev/null 2024-04-25T08:41:28.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:28.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj21 /dev/null 2024-04-25T08:41:28.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:28.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj22 /dev/null 2024-04-25T08:41:28.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:28.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj23 /dev/null 2024-04-25T08:41:28.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:28.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj24 /dev/null 2024-04-25T08:41:28.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:28.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj25 /dev/null 2024-04-25T08:41:28.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:28.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj26 /dev/null 2024-04-25T08:41:28.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:28.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj27 /dev/null 2024-04-25T08:41:29.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:29.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj28 /dev/null 2024-04-25T08:41:29.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:29.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj29 /dev/null 2024-04-25T08:41:29.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:29.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj30 /dev/null 2024-04-25T08:41:29.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:29.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj31 /dev/null 2024-04-25T08:41:29.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:29.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj32 /dev/null 2024-04-25T08:41:29.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:29.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj33 /dev/null 2024-04-25T08:41:29.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:29.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj34 /dev/null 2024-04-25T08:41:30.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:30.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj35 /dev/null 2024-04-25T08:41:30.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:30.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj36 /dev/null 2024-04-25T08:41:30.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:30.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj37 /dev/null 2024-04-25T08:41:30.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:30.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj38 /dev/null 2024-04-25T08:41:30.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:30.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj39 /dev/null 2024-04-25T08:41:30.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:30.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj40 /dev/null 2024-04-25T08:41:30.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:30.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj41 /dev/null 2024-04-25T08:41:31.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:31.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj42 /dev/null 2024-04-25T08:41:31.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:31.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj43 /dev/null 2024-04-25T08:41:31.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:31.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj44 /dev/null 2024-04-25T08:41:31.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:31.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj45 /dev/null 2024-04-25T08:41:31.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:31.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj46 /dev/null 2024-04-25T08:41:31.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:31.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj47 /dev/null 2024-04-25T08:41:31.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:31.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj48 /dev/null 2024-04-25T08:41:32.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:32.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj49 /dev/null 2024-04-25T08:41:32.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:32.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj50 /dev/null 2024-04-25T08:41:32.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:32.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj51 /dev/null 2024-04-25T08:41:32.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:32.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj52 /dev/null 2024-04-25T08:41:32.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:32.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj53 /dev/null 2024-04-25T08:41:32.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:32.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj54 /dev/null 2024-04-25T08:41:33.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:33.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj55 /dev/null 2024-04-25T08:41:33.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:33.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj56 /dev/null 2024-04-25T08:41:33.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:33.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj57 /dev/null 2024-04-25T08:41:33.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:33.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj58 /dev/null 2024-04-25T08:41:33.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:33.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj59 /dev/null 2024-04-25T08:41:33.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:33.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj60 /dev/null 2024-04-25T08:41:33.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:33.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj61 /dev/null 2024-04-25T08:41:34.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:34.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj62 /dev/null 2024-04-25T08:41:34.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:34.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj63 /dev/null 2024-04-25T08:41:34.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:34.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj64 /dev/null 2024-04-25T08:41:34.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:34.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj65 /dev/null 2024-04-25T08:41:34.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:34.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj66 /dev/null 2024-04-25T08:41:34.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:34.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj67 /dev/null 2024-04-25T08:41:34.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:34.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj68 /dev/null 2024-04-25T08:41:34.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:34.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj69 /dev/null 2024-04-25T08:41:35.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:35.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj70 /dev/null 2024-04-25T08:41:35.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:35.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj71 /dev/null 2024-04-25T08:41:35.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:35.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj72 /dev/null 2024-04-25T08:41:35.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:35.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj73 /dev/null 2024-04-25T08:41:35.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:35.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj74 /dev/null 2024-04-25T08:41:35.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:35.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj75 /dev/null 2024-04-25T08:41:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:35.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj76 /dev/null 2024-04-25T08:41:36.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:36.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj77 /dev/null 2024-04-25T08:41:36.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:36.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj78 /dev/null 2024-04-25T08:41:36.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:36.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj79 /dev/null 2024-04-25T08:41:36.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:36.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj80 /dev/null 2024-04-25T08:41:36.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:36.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj81 /dev/null 2024-04-25T08:41:36.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:36.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj82 /dev/null 2024-04-25T08:41:37.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:37.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj83 /dev/null 2024-04-25T08:41:37.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:37.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj84 /dev/null 2024-04-25T08:41:37.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:37.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj85 /dev/null 2024-04-25T08:41:37.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:37.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj86 /dev/null 2024-04-25T08:41:37.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:37.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj87 /dev/null 2024-04-25T08:41:37.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:37.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj88 /dev/null 2024-04-25T08:41:37.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:37.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj89 /dev/null 2024-04-25T08:41:37.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:37.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj90 /dev/null 2024-04-25T08:41:38.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:38.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj91 /dev/null 2024-04-25T08:41:38.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:38.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj92 /dev/null 2024-04-25T08:41:38.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:38.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj93 /dev/null 2024-04-25T08:41:38.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:38.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj94 /dev/null 2024-04-25T08:41:38.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:38.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj95 /dev/null 2024-04-25T08:41:38.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:38.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj96 /dev/null 2024-04-25T08:41:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:39.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj97 /dev/null 2024-04-25T08:41:39.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:39.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj98 /dev/null 2024-04-25T08:41:39.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:39.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj99 /dev/null 2024-04-25T08:41:39.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:41:39.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj100 /dev/null 2024-04-25T08:41:39.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: cat td/osd-recovery-stats/osd.1.pid 2024-04-25T08:41:39.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: kill 310110 2024-04-25T08:41:39.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:474: TEST_recovery_multi: ceph osd down osd.1 2024-04-25T08:41:40.234 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already down. 2024-04-25T08:41:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:475: TEST_recovery_multi: activate_osd td/osd-recovery-stats 0 2024-04-25T08:41:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-04-25T08:41:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:41:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T08:41:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:41:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:41:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:41:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:41:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:41:40.254 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:41:40.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:41:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:41:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:41:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:41:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:41:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:41:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:41:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:41:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:41:40.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:41:40.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:41:40.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:41:40.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T08:41:40.257 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:41:40.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:41:40.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/0/whoami 2024-04-25T08:41:40.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T08:41:40.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:41:40.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:41:40.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:41:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:41:40.291+0000 7f55227d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:40.371 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:41:40.379+0000 7f55227d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:40.372 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:41:40.379+0000 7f55227d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:40.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T08:41:40.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:41:40.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:41:40.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:41:40.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:41:40.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:40.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:41:40.660 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:41:40.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:40.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:41:40.908 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:41:40.915+0000 7f55227d87c0 -1 Falling back to public interface 2024-04-25T08:41:41.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:42.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:42.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:42.072 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:41:42.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:41:42.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:42.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:41:42.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:43.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:43.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:43.486 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:41:43.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:41:43.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:43.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:41:44.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:44.379 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:41:44.387+0000 7f55227d87c0 -1 osd.0 40 log_to_monitors true 2024-04-25T08:41:45.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:45.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:45.058 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:41:45.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:41:45.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:45.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:41:45.598 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 49 up_thru 49 down_at 44 last_clean_interval [5,40) [v2:127.0.0.1:6802/776107050,v1:127.0.0.1:6803/776107050] [v2:127.0.0.1:6804/776107050,v1:127.0.0.1:6805/776107050] exists,up 38dbd4be-4ad1-4d36-ab43-b04d4b907711 2024-04-25T08:41:45.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:41:45.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:41:45.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:41:45.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:476: TEST_recovery_multi: sleep 3 2024-04-25T08:41:48.602 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: expr 100 + 1 2024-04-25T08:41:48.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: seq 101 200 2024-04-25T08:41:48.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:48.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj101 /dev/null 2024-04-25T08:41:48.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:48.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj102 /dev/null 2024-04-25T08:41:48.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:48.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj103 /dev/null 2024-04-25T08:41:49.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:49.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj104 /dev/null 2024-04-25T08:41:49.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:49.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj105 /dev/null 2024-04-25T08:41:49.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:49.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj106 /dev/null 2024-04-25T08:41:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj107 /dev/null 2024-04-25T08:41:49.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:49.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj108 /dev/null 2024-04-25T08:41:49.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:49.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj109 /dev/null 2024-04-25T08:41:49.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:49.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj110 /dev/null 2024-04-25T08:41:49.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj111 /dev/null 2024-04-25T08:41:50.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:50.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj112 /dev/null 2024-04-25T08:41:50.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:50.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj113 /dev/null 2024-04-25T08:41:50.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:50.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj114 /dev/null 2024-04-25T08:41:50.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:50.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj115 /dev/null 2024-04-25T08:41:50.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:50.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj116 /dev/null 2024-04-25T08:41:50.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:50.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj117 /dev/null 2024-04-25T08:41:50.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:50.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj118 /dev/null 2024-04-25T08:41:51.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:51.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj119 /dev/null 2024-04-25T08:41:51.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:51.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj120 /dev/null 2024-04-25T08:41:51.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:51.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj121 /dev/null 2024-04-25T08:41:51.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:51.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj122 /dev/null 2024-04-25T08:41:51.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:51.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj123 /dev/null 2024-04-25T08:41:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:51.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj124 /dev/null 2024-04-25T08:41:51.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:51.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj125 /dev/null 2024-04-25T08:41:51.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:51.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj126 /dev/null 2024-04-25T08:41:52.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:52.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj127 /dev/null 2024-04-25T08:41:52.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:52.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj128 /dev/null 2024-04-25T08:41:52.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:52.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj129 /dev/null 2024-04-25T08:41:52.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:52.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj130 /dev/null 2024-04-25T08:41:52.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:52.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj131 /dev/null 2024-04-25T08:41:52.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:52.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj132 /dev/null 2024-04-25T08:41:52.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:52.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj133 /dev/null 2024-04-25T08:41:53.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:53.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj134 /dev/null 2024-04-25T08:41:53.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:53.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj135 /dev/null 2024-04-25T08:41:53.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:53.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj136 /dev/null 2024-04-25T08:41:53.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:53.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj137 /dev/null 2024-04-25T08:41:53.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:53.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj138 /dev/null 2024-04-25T08:41:53.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:53.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj139 /dev/null 2024-04-25T08:41:53.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:53.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj140 /dev/null 2024-04-25T08:41:53.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:53.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj141 /dev/null 2024-04-25T08:41:54.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:54.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj142 /dev/null 2024-04-25T08:41:54.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:54.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj143 /dev/null 2024-04-25T08:41:54.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:54.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj144 /dev/null 2024-04-25T08:41:54.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:54.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj145 /dev/null 2024-04-25T08:41:54.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:54.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj146 /dev/null 2024-04-25T08:41:54.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:54.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj147 /dev/null 2024-04-25T08:41:54.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:54.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj148 /dev/null 2024-04-25T08:41:55.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:55.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj149 /dev/null 2024-04-25T08:41:55.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:55.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj150 /dev/null 2024-04-25T08:41:55.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:55.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj151 /dev/null 2024-04-25T08:41:55.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:55.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj152 /dev/null 2024-04-25T08:41:55.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:55.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj153 /dev/null 2024-04-25T08:41:55.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:55.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj154 /dev/null 2024-04-25T08:41:55.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:55.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj155 /dev/null 2024-04-25T08:41:55.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:55.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj156 /dev/null 2024-04-25T08:41:56.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:56.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj157 /dev/null 2024-04-25T08:41:56.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:56.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj158 /dev/null 2024-04-25T08:41:56.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:56.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj159 /dev/null 2024-04-25T08:41:56.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:56.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj160 /dev/null 2024-04-25T08:41:56.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:56.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj161 /dev/null 2024-04-25T08:41:56.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:56.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj162 /dev/null 2024-04-25T08:41:56.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:56.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj163 /dev/null 2024-04-25T08:41:57.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:57.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj164 /dev/null 2024-04-25T08:41:57.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:57.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj165 /dev/null 2024-04-25T08:41:57.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:57.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj166 /dev/null 2024-04-25T08:41:57.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:57.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj167 /dev/null 2024-04-25T08:41:57.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:57.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj168 /dev/null 2024-04-25T08:41:57.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:57.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj169 /dev/null 2024-04-25T08:41:57.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:57.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj170 /dev/null 2024-04-25T08:41:57.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:57.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj171 /dev/null 2024-04-25T08:41:58.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:58.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj172 /dev/null 2024-04-25T08:41:58.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:58.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj173 /dev/null 2024-04-25T08:41:58.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:58.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj174 /dev/null 2024-04-25T08:41:58.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:58.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj175 /dev/null 2024-04-25T08:41:58.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:58.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj176 /dev/null 2024-04-25T08:41:58.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:58.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj177 /dev/null 2024-04-25T08:41:58.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:58.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj178 /dev/null 2024-04-25T08:41:59.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:59.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj179 /dev/null 2024-04-25T08:41:59.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:59.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj180 /dev/null 2024-04-25T08:41:59.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:59.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj181 /dev/null 2024-04-25T08:41:59.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:59.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj182 /dev/null 2024-04-25T08:41:59.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:59.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj183 /dev/null 2024-04-25T08:41:59.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:59.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj184 /dev/null 2024-04-25T08:41:59.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:59.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj185 /dev/null 2024-04-25T08:41:59.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:41:59.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj186 /dev/null 2024-04-25T08:42:00.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:00.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj187 /dev/null 2024-04-25T08:42:00.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:00.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj188 /dev/null 2024-04-25T08:42:00.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:00.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj189 /dev/null 2024-04-25T08:42:00.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:00.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj190 /dev/null 2024-04-25T08:42:00.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:00.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj191 /dev/null 2024-04-25T08:42:00.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:00.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj192 /dev/null 2024-04-25T08:42:00.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:00.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj193 /dev/null 2024-04-25T08:42:01.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:01.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj194 /dev/null 2024-04-25T08:42:01.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:01.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj195 /dev/null 2024-04-25T08:42:01.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:01.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj196 /dev/null 2024-04-25T08:42:01.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:01.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj197 /dev/null 2024-04-25T08:42:01.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:01.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj198 /dev/null 2024-04-25T08:42:01.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:01.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj199 /dev/null 2024-04-25T08:42:01.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:42:01.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj200 /dev/null 2024-04-25T08:42:01.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: get_pg test obj1 2024-04-25T08:42:01.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:42:01.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:42:01.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:42:01.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:42:02.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: local PG=1.0 2024-04-25T08:42:02.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: get_not_primary test obj200 2024-04-25T08:42:02.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:42:02.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj200 2024-04-25T08:42:02.389 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj200 2024-04-25T08:42:02.389 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:42:02.390 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj200 2024-04-25T08:42:02.390 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj200 2024-04-25T08:42:02.390 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:42:02.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=0 2024-04-25T08:42:02.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 0)) | .[0]' 2024-04-25T08:42:02.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj200 2024-04-25T08:42:03.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: local otherosd=2 2024-04-25T08:42:03.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:486: TEST_recovery_multi: ceph osd unset noout 2024-04-25T08:42:04.010 INFO:tasks.workunit.client.0.smithi102.stderr:noout is unset 2024-04-25T08:42:04.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:487: TEST_recovery_multi: ceph osd out osd.1 osd.2 2024-04-25T08:42:04.725 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already out. osd.2 is already out. 2024-04-25T08:42:04.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:488: TEST_recovery_multi: activate_osd td/osd-recovery-stats 1 2024-04-25T08:42:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-04-25T08:42:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:42:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T08:42:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:42:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:42:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:42:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:42:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:42:04.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:42:04.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:42:04.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:42:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:42:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:42:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:42:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:42:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:42:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:42:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:42:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:42:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:42:04.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:42:04.750 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:42:04.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T08:42:04.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:42:04.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/1/whoami 2024-04-25T08:42:04.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T08:42:04.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:42:04.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:42:04.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:42:04.778 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:42:04.787+0000 7f2217fb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:42:04.781 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:42:04.787+0000 7f2217fb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:42:04.782 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:42:04.791+0000 7f2217fb67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:42:05.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T08:42:05.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:42:05.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:42:05.145 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:42:05.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:42:05.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:42:05.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:05.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:42:05.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:05.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:05.327 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:42:05.335+0000 7f2217fb67c0 -1 Falling back to public interface 2024-04-25T08:42:05.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:06.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:06.585 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:42:06.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:42:06.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:06.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:07.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:08.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:08.325 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:42:08.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:08.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:42:08.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:08.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:08.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:09.055 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:42:09.063+0000 7f2217fb67c0 -1 osd.1 45 log_to_monitors true 2024-04-25T08:42:09.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:09.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:09.758 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:42:09.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:42:09.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:09.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:10.215 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up out weight 0 up_from 58 up_thru 44 down_at 46 last_clean_interval [10,45) [v2:127.0.0.1:6810/2504960979,v1:127.0.0.1:6811/2504960979] [v2:127.0.0.1:6812/2504960979,v1:127.0.0.1:6813/2504960979] exists,up 2c7386c4-296c-4b50-afff-2a1635217bfa 2024-04-25T08:42:10.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:42:10.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:42:10.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:42:10.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:489: TEST_recovery_multi: sleep 3 2024-04-25T08:42:13.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:491: TEST_recovery_multi: ceph osd pool set test size 4 2024-04-25T08:42:14.147 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 4 2024-04-25T08:42:14.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:492: TEST_recovery_multi: ceph osd unset norecover 2024-04-25T08:42:15.981 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is unset 2024-04-25T08:42:16.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:42:16.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:42:16.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:42:16.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:42:16.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:42:16.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T08:42:16.890 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:42:16.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:494: TEST_recovery_multi: sleep 2 2024-04-25T08:42:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:496: TEST_recovery_multi: wait_for_clean 2024-04-25T08:42:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:42:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:42:18.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:42:18.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:42:18.910 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:42:18.910 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:42:18.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:42:18.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:42:18.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:42:19.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:42:19.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:42:19.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:42:19.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:42:19.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:42:19.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:42:19.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:42:19.485 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:42:19.485 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:42:19.485 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:42:19.485 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T08:42:19.485 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T08:42:19.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:42:19.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:19.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:42:19.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=210453397513 2024-04-25T08:42:19.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 210453397513 2024-04-25T08:42:19.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513' 2024-04-25T08:42:19.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:19.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:42:19.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=249108103173 2024-04-25T08:42:19.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 249108103173 2024-04-25T08:42:19.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513 1-249108103173' 2024-04-25T08:42:19.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:19.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:42:20.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509492 2024-04-25T08:42:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509492 2024-04-25T08:42:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513 1-249108103173 2-64424509492' 2024-04-25T08:42:20.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:20.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:42:20.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345957 2024-04-25T08:42:20.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345957 2024-04-25T08:42:20.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513 1-249108103173 2-64424509492 3-85899345957' 2024-04-25T08:42:20.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:20.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:42:20.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182434 2024-04-25T08:42:20.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182434 2024-04-25T08:42:20.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513 1-249108103173 2-64424509492 3-85899345957 4-107374182434' 2024-04-25T08:42:20.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:42:20.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:42:20.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018905 2024-04-25T08:42:20.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018905 2024-04-25T08:42:20.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513 1-249108103173 2-64424509492 3-85899345957 4-107374182434 5-128849018905' 2024-04-25T08:42:20.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:20.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-210453397513 2024-04-25T08:42:20.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:20.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:42:20.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-210453397513 2024-04-25T08:42:20.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:20.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=210453397513 2024-04-25T08:42:20.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 210453397513' 2024-04-25T08:42:20.579 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 210453397513 2024-04-25T08:42:20.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:42:21.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 210453397514 -lt 210453397513 2024-04-25T08:42:21.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:21.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-249108103173 2024-04-25T08:42:21.075 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:21.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:42:21.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-249108103173 2024-04-25T08:42:21.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:21.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=249108103173 2024-04-25T08:42:21.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 249108103173' 2024-04-25T08:42:21.078 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 249108103173 2024-04-25T08:42:21.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:42:21.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 249108103173 -lt 249108103173 2024-04-25T08:42:21.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:21.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509492 2024-04-25T08:42:21.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:21.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:42:21.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:21.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509492 2024-04-25T08:42:21.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509492 2024-04-25T08:42:21.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509492' 2024-04-25T08:42:21.502 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509492 2024-04-25T08:42:21.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:42:22.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509492 -lt 64424509492 2024-04-25T08:42:22.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:22.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345957 2024-04-25T08:42:22.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:22.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:42:22.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345957 2024-04-25T08:42:22.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:22.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345957 2024-04-25T08:42:22.067 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345957 2024-04-25T08:42:22.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345957' 2024-04-25T08:42:22.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:42:22.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345957 -lt 85899345957 2024-04-25T08:42:22.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:22.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182434 2024-04-25T08:42:22.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:22.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:42:22.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182434 2024-04-25T08:42:22.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:22.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182434 2024-04-25T08:42:22.768 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182434 2024-04-25T08:42:22.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182434' 2024-04-25T08:42:22.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:42:23.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182434 -lt 107374182434 2024-04-25T08:42:23.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:42:23.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018905 2024-04-25T08:42:23.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:42:23.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:42:23.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018905 2024-04-25T08:42:23.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:42:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018905 2024-04-25T08:42:23.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018905' 2024-04-25T08:42:23.371 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018905 2024-04-25T08:42:23.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:42:23.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018905 -lt 128849018905 2024-04-25T08:42:23.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:42:23.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:23.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:24.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:42:24.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:24.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:24.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:24.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:24.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:24.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:24.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:24.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:24.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:24.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:24.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:25.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:25.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:42:25.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:42:25.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:42:25.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:25.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:25.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:25.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:25.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:25.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:25.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:25.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:25.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:25.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:25.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:25.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:25.924 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:25.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:26.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:26.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:26.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:26.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:26.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:26.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:26.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:26.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:26.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:27.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:42:27.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:42:27.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:27.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:27.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:27.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:27.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:27.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:27.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:27.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:27.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:27.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:27.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:27.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:27.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:27.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:27.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:28.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:28.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:28.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:28.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:28.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:28.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:28.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:28.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:28.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:29.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:42:29.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:42:29.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:29.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:29.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:29.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:29.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:29.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:29.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:29.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:29.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:29.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:29.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:29.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:29.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:29.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:29.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:30.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:30.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:30.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:30.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:30.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:30.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:30.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:30.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:30.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:30.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:42:30.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:42:30.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:30.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:30.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:31.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:31.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:31.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:31.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:31.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:31.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:31.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:31.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:31.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:31.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:31.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:31.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:32.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:32.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:32.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:32.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:32.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:32.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:32.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:32.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:32.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:32.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:42:32.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:42:32.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:32.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:32.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:32.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:32.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:32.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:32.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:32.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:32.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:32.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:32.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:33.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:33.229 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:33.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:33.230 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:33.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:33.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:33.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:33.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:33.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:33.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:33.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:33.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:33.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:34.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:42:34.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:42:34.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:34.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:34.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:34.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:34.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:34.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:34.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:34.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:34.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:34.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:34.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:35.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:35.168 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:35.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:35.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:35.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:35.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:35.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:35.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:35.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:35.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:35.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:35.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:35.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:36.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:42:36.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:42:36.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:36.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:36.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:36.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:36.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:36.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:36.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:36.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:36.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:36.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:36.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:37.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:37.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:37.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:37.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:37.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:37.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:37.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:37.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:37.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:37.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:37.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:37.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:37.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:42:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:42:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:38.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:38.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:38.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:38.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:38.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:38.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:38.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:38.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:38.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:38.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:38.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:38.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:39.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:39.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:39.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:39.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:39.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:39.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:40.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:42:40.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:42:40.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:40.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:40.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:40.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:40.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:40.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:40.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:40.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:40.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:40.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:40.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:40.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:40.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:40.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:40.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:41.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:41.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:41.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:41.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:41.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:41.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:41.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:41.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:41.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:41.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:42:41.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:42:41.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:41.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:41.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:42.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:42.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:42.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:42.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:42.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:42.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:42.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:42.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:42.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:42.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:42.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:42.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:43.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:43.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:43.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:43.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:43.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:43.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:43.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:43.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:43.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:43.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:42:43.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:42:43.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:43.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:43.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:43.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:43.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:43.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:43.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:43.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:43.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:43.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:43.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:44.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:44.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:44.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:44.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:44.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:44.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:44.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:44.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:44.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:44.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:44.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:44.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:44.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:45.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:42:45.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:42:45.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:45.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:45.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:45.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:45.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:45.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:45.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:45.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:45.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:45.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:45.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:46.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:46.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:46.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:46.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:46.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:46.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:46.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:46.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:46.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:46.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:46.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:46.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:46.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:47.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:42:47.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:42:47.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:47.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:47.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:47.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:47.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:47.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:47.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:47.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:47.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:47.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:47.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:48.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:48.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:48.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:48.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:48.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:48.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:48.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:48.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:48.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:48.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:48.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:48.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:48.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:49.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:42:49.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:42:49.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:49.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:49.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:49.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:49.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:49.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:49.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:49.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:49.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:49.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:49.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:49.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:49.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:49.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:49.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:50.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:50.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:50.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:50.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:50.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:50.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:50.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:50.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:50.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:51.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:42:51.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:42:51.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:51.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:51.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:51.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:51.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:51.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:51.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:51.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:51.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:51.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:51.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:51.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:51.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:51.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:51.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:52.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:52.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:52.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:52.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:52.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:52.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:52.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:52.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:52.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:52.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:42:52.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:42:52.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:52.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:52.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:53.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:53.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:53.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:53.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:53.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:53.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:53.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:53.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:53.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:53.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:53.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:53.437 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:53.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:53.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:53.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:53.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:53.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:53.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:53.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:53.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:53.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:54.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:42:54.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:42:54.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:54.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:54.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:54.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:54.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:54.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:54.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:54.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:54.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:54.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:54.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:55.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:42:55.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:55.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:55.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:55.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:42:55.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:42:55.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:42:55.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:42:55.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:42:55.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:42:55.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:42:55.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:42:55.558 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:42:56.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:42:56.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:42:56.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:42:56.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:42:56.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:42:56.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:42:56.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:42:56.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:42:56.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:42:56.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:42:56.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:42:56.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:42:56.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:42:56.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:42:56.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:42:56.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:42:56.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:42:57.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:42:57.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:42:57.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:42:57.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:42:57.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:42:57.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:42:57.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:42:57.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:42:57.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: primary=0 2024-04-25T08:42:57.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:501: TEST_recovery_multi: local log=td/osd-recovery-stats/osd.0.log 2024-04-25T08:42:57.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:502: TEST_recovery_multi: check td/osd-recovery-stats 1.0 0 replicated 399 0 300 0 99 0 2024-04-25T08:42:57.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:42:57.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:42:57.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=0 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=399 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=300 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=99 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.0.log 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:42:57.655 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:42:57.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[0,4,3,5]/[0,3]' 2024-04-25T08:42:57.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:42:57.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:42:57.687 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:42:57.687 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:42:57.687 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:42:57.687 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:42:57.687 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:42:57.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=399 2024-04-25T08:42:57.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 399 399 2024-04-25T08:42:57.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=399 2024-04-25T08:42:57.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:42:57.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=399 2024-04-25T08:42:57.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:42:57.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:42:57.693 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:42:57.693 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:42:57.693 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:42:57.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:42:57.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:42:57.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:42:57.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:42:57.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:42:57.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:42:57.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:42:57.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:42:57.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:42:57.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:42:57.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:42:57.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:42:57.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:42:57.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=300 2024-04-25T08:42:57.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 300 300 2024-04-25T08:42:57.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=300 2024-04-25T08:42:57.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:42:57.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=300 2024-04-25T08:42:57.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:42:57.712 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:42:57.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:42:57.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:42:57.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:42:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:42:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:42:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:42:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:42:57.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:42:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:42:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 99 ']' 2024-04-25T08:42:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 0' 2024-04-25T08:42:57.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:42:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:42:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:42:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-04-25T08:42:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-04-25T08:42:57.726 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:42:57.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=99 2024-04-25T08:42:57.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 99 99 2024-04-25T08:42:57.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=99 2024-04-25T08:42:57.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:42:57.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=99 2024-04-25T08:42:57.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:42:57.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:42:57.732 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:42:57.732 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-04-25T08:42:57.732 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:42:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-04-25T08:42:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-04-25T08:42:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:42:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:42:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:42:57.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:42:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:504: TEST_recovery_multi: delete_pool test 2024-04-25T08:42:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:42:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:42:58.268 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T08:42:58.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:505: TEST_recovery_multi: kill_daemons td/osd-recovery-stats 2024-04-25T08:42:58.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:42:58.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:42:58.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:42:58.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:42:58.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:42:59.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:42:59.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:42:59.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:42:59.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:42:59.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:42:59.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:42:59.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:42:59.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:42:59.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:42:59.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:42:59.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:42:59.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:42:59.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:42:59.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:42:59.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:42:59.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:42:59.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:42:59.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:42:59.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:42:59.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:42:59.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:42:59.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:42:59.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:42:59.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:42:59.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:42:59.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:42:59.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:42:59.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:42:59.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:42:59.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:42:59.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:42:59.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:42:59.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:42:59.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:42:59.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:42:59.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:42:59.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:42:59.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:42:59.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:42:59.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:42:59.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:42:59.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:42:59.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:42:59.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:42:59.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:42:59.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:42:59.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:42:59.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:42:59.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:42:59.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:42:59.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:42:59.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:42:59.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:42:59.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:42:59.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:42:59.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:42:59.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:42:59.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:42:59.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:42:59.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:42:59.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:42:59.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:42:59.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:42:59.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:42:59.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:42:59.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:42:59.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.277516 2024-04-25T08:42:59.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:42:59.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:42:59.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:42:59.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:42:59.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_replicated_out1 td/osd-recovery-stats 2024-04-25T08:42:59.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:167: TEST_recovery_replicated_out1: local dir=td/osd-recovery-stats 2024-04-25T08:42:59.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:169: TEST_recovery_replicated_out1: do_recovery_out1 td/osd-recovery-stats replicated 2024-04-25T08:42:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-04-25T08:42:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-04-25T08:42:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=replicated 2024-04-25T08:42:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-04-25T08:42:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:42:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:42:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:42:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:42:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:42:59.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:43:00.174 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:43:00.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:00.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:00.175 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:00.175 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:00.175 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:43:00.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:43:00.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:43:00.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:43:00.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:43:00.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:43:00.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:43:00.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:43:00.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:43:00.396 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:43:00.396 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:43:00.396 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:43:00.396 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:43:00.396 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:00.397 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:43:00.397 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:43:00.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:43:00.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get fsid 2024-04-25T08:43:00.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:43:00.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:43:00.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:43:00.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:43:00.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:43:00.513 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:43:00.513 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:43:00.513 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:43:00.513 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:43:00.513 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:00.513 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:43:00.513 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:43:00.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:43:00.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get mon_host 2024-04-25T08:43:00.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-04-25T08:43:00.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:43:00.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:43:00.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:43:00.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:43:00.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:43:00.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:43:01.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:43:01.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:01.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:01.550 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:01.550 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:01.550 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:43:01.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:43:01.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:43:01.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:43:01.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-04-25T08:43:01.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:43:01.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:43:01.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:01.587 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:43:01.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:43:01.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:43:01.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae438758-3c0f-40c0-9ed1-4cc43c272fc9 2024-04-25T08:43:01.589 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 ae438758-3c0f-40c0-9ed1-4cc43c272fc9 2024-04-25T08:43:01.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ae438758-3c0f-40c0-9ed1-4cc43c272fc9' 2024-04-25T08:43:01.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:43:01.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVFypmG5uvJBAAnGjUxK54Ho2OaUlZMk7Rzg== 2024-04-25T08:43:01.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVFypmG5uvJBAAnGjUxK54Ho2OaUlZMk7Rzg=="}' 2024-04-25T08:43:01.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae438758-3c0f-40c0-9ed1-4cc43c272fc9 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:43:01.922 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:43:01.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:43:01.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCVFypmG5uvJBAAnGjUxK54Ho2OaUlZMk7Rzg== --osd-uuid ae438758-3c0f-40c0-9ed1-4cc43c272fc9 2024-04-25T08:43:01.978 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:01.986+0000 7fa57299c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:02.000 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:02.006+0000 7fa57299c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:02.002 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:02.010+0000 7fa57299c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:02.003 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:02.010+0000 7fa57299c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:43:02.003 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:02.010+0000 7fa57299c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:43:02.003 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:02.010+0000 7fa57299c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:43:02.047 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:02.054+0000 7fa57299c7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:43:08.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:43:08.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:43:08.553 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:43:08.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:43:08.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:43:09.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:43:09.094 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:43:09.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:43:09.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:43:09.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:43:09.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:43:09.126 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:09.134+0000 7fef2f8417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:09.129 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:09.134+0000 7fef2f8417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:09.130 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:09.138+0000 7fef2f8417c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:09.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:43:09.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:43:09.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:43:09.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:43:09.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:43:09.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:09.490 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:43:09.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:43:09.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:09.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:09.679 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:09.686+0000 7fef2f8417c0 -1 Falling back to public interface 2024-04-25T08:43:09.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:10.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:10.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:10.926 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:43:10.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:43:10.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:10.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:11.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:12.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:12.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:12.340 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:43:12.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:43:12.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:12.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:12.367 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:12.374+0000 7fef2f8417c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:43:12.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:13.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:13.909 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:43:13.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:13.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:43:13.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:13.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:14.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:15.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:15.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:15.358 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:43:15.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:43:15.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:15.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:15.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:16.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:16.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:16.826 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:43:16.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:43:16.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:16.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:17.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:18.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:18.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:18.493 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:43:18.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:43:18.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:18.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:18.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:19.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:19.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:19.939 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:43:19.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:43:19.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:19.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:20.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:21.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:21.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:21.417 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:43:21.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:43:21.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:21.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:21.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:22.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:22.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:22.877 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:43:22.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:43:22.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:22.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:23.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:24.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:24.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:24.474 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:43:24.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:43:24.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:24.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:24.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:25.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:25.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:25.928 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:43:25.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:43:25.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:25.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:27.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:28.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:28.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:28.331 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:43:28.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:43:28.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:28.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:29.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:30.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:30.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:30.205 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:43:30.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:43:30.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:30.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:30.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:31.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:31.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:31.730 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:43:31.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:43:31.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:31.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:32.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:33.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:33.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:33.537 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:43:33.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:43:33.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:33.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:34.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:35.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:35.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:35.801 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:43:35.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:43:35.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:35.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:36.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:37.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:37.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:37.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:43:37.687 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:43:37.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:37.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:43:38.179 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1455138490,v1:127.0.0.1:6803/1455138490] [v2:127.0.0.1:6804/1455138490,v1:127.0.0.1:6805/1455138490] exists,up ae438758-3c0f-40c0-9ed1-4cc43c272fc9 2024-04-25T08:43:38.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:43:38.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:43:38.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:43:38.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-04-25T08:43:38.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:43:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:43:38.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:43:38.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:43:38.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:43:38.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:43:38.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:43:38.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:43:38.185 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:43:38.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9118ca50-0870-40f3-ad1e-2eb70c045262 2024-04-25T08:43:38.186 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 9118ca50-0870-40f3-ad1e-2eb70c045262 2024-04-25T08:43:38.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9118ca50-0870-40f3-ad1e-2eb70c045262' 2024-04-25T08:43:38.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:43:38.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6Fypm4drJDBAAkWZ8LFOgruln7XdeWthTmg== 2024-04-25T08:43:38.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6Fypm4drJDBAAkWZ8LFOgruln7XdeWthTmg=="}' 2024-04-25T08:43:38.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9118ca50-0870-40f3-ad1e-2eb70c045262 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:43:38.701 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:43:38.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:43:38.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6Fypm4drJDBAAkWZ8LFOgruln7XdeWthTmg== --osd-uuid 9118ca50-0870-40f3-ad1e-2eb70c045262 2024-04-25T08:43:38.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:38.749+0000 7f0ccdd6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:38.821 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:38.829+0000 7f0ccdd6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:38.823 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:38.829+0000 7f0ccdd6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:38.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:38.829+0000 7f0ccdd6a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:43:38.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:38.829+0000 7f0ccdd6a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:43:38.824 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:38.829+0000 7f0ccdd6a7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:43:38.948 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:38.953+0000 7f0ccdd6a7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:43:46.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:43:46.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:43:46.014 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T08:43:46.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:43:46.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:43:46.573 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:43:46.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:43:46.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:43:46.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:43:46.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:43:46.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:43:46.605 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:46.613+0000 7fa245df27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:46.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:46.613+0000 7fa245df27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:46.608 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:46.613+0000 7fa245df27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:47.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:43:47.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:43:47.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:43:47.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:43:47.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:43:47.018 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:43:47.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:47.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:43:47.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:47.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:43:47.170 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:47.177+0000 7fa245df27c0 -1 Falling back to public interface 2024-04-25T08:43:47.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:48.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:48.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:48.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:43:48.458 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:43:48.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:48.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:43:48.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:49.787 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:43:49.793+0000 7fa245df27c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:43:49.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:49.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:49.866 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:43:49.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:43:49.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:49.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:43:50.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:51.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:51.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:51.615 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:43:51.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:43:51.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:51.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:43:52.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:53.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:53.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:53.037 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:43:53.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:43:53.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:53.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:43:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:54.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:54.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:54.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:43:54.484 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:43:54.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:54.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:43:55.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:56.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:56.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:56.094 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:43:56.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:43:56.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:56.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:43:56.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:57.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:57.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:57.667 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:43:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:43:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:43:58.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:59.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:59.111 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:43:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:43:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:59.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:43:59.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:00.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:00.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:00.542 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:44:00.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:44:00.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:00.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:01.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:02.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:02.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:02.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:44:02.093 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:44:02.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:02.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:02.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:03.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:03.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:03.732 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:44:03.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:44:03.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:03.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:04.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:05.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:05.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:05.353 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:44:05.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:44:05.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:05.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:06.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:07.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:07.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:07.294 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:44:07.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:44:07.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:07.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:08.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:09.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:09.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:09.076 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:44:09.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:44:09.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:09.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:10.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:10.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:10.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:44:10.948 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:44:10.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:10.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:12.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:13.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:13.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:13.192 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:44:13.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:44:13.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:13.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:13.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:14.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:14.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:14.991 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:44:14.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:44:14.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:14.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:15.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:16.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:16.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:16.618 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:44:16.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:44:16.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:16.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:44:17.090 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2145452161,v1:127.0.0.1:6811/2145452161] [v2:127.0.0.1:6812/2145452161,v1:127.0.0.1:6813/2145452161] exists,up 9118ca50-0870-40f3-ad1e-2eb70c045262 2024-04-25T08:44:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:44:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:44:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:44:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-04-25T08:44:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:44:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:44:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:44:17.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:44:17.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:44:17.094 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:44:17.094 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:44:17.094 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:44:17.095 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:44:17.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:44:17.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=82af365f-20c6-4b19-8cb3-1af302acdb20 2024-04-25T08:44:17.097 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 82af365f-20c6-4b19-8cb3-1af302acdb20 2024-04-25T08:44:17.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 82af365f-20c6-4b19-8cb3-1af302acdb20' 2024-04-25T08:44:17.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:44:17.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhFypmLliIBxAAvd4ZWVWNMxZS8a4yOXQY0A== 2024-04-25T08:44:17.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhFypmLliIBxAAvd4ZWVWNMxZS8a4yOXQY0A=="}' 2024-04-25T08:44:17.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 82af365f-20c6-4b19-8cb3-1af302acdb20 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:44:17.589 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:44:17.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:44:17.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhFypmLliIBxAAvd4ZWVWNMxZS8a4yOXQY0A== --osd-uuid 82af365f-20c6-4b19-8cb3-1af302acdb20 2024-04-25T08:44:17.640 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:17.644+0000 7fa08097b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:17.683 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:17.688+0000 7fa08097b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:17.684 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:17.688+0000 7fa08097b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:17.684 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:17.688+0000 7fa08097b7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:44:17.684 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:17.688+0000 7fa08097b7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:44:17.685 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:17.688+0000 7fa08097b7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:44:17.714 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:17.720+0000 7fa08097b7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:44:24.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:44:24.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:44:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:44:24.681 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T08:44:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:44:25.217 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T08:44:25.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:44:25.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:44:25.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:44:25.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:44:25.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:44:25.243 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:25.248+0000 7fdf476827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:25.245 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:25.252+0000 7fdf476827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:25.246 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:25.252+0000 7fdf476827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:44:25.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:44:25.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:44:25.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:44:25.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:44:25.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:25.654 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:44:25.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:44:25.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:25.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:25.814 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:25.820+0000 7fdf476827c0 -1 Falling back to public interface 2024-04-25T08:44:26.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:27.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:27.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:27.100 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:44:27.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:44:27.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:27.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:27.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:28.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:28.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:28.527 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:44:28.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:44:28.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:28.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:28.589 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:28.592+0000 7fdf476827c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:44:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:30.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:30.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:30.025 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:44:30.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:44:30.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:30.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:30.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:31.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:31.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:31.530 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:44:31.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:44:31.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:31.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:32.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:33.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:33.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:33.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:44:33.095 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:44:33.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:33.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:33.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:34.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:34.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:34.700 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:44:34.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:44:34.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:34.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:35.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:36.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:36.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:36.274 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:44:36.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:44:36.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:36.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:36.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:37.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:37.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:37.763 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:44:37.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:44:37.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:37.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:38.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:39.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:39.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:39.232 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:44:39.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:44:39.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:39.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:39.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:40.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:40.685 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:44:40.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:44:40.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:40.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:41.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:42.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:42.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:42.259 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:44:42.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:44:42.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:42.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:42.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:43.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:43.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:43.789 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:44:43.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:44:43.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:43.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:44.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:45.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:45.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:45.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:44:45.648 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:44:45.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:45.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:46.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:47.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:47.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:47.470 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:44:47.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:44:47.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:47.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:48.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:49.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:49.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:49.279 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:44:49.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:44:49.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:49.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:50.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:51.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:51.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:51.148 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:44:51.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:44:51.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:51.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:52.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:53.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:53.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:53.059 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:44:53.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:44:53.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:53.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:54.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:55.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:55.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:55.065 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:44:55.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:44:55.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:55.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:44:55.639 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/516459931,v1:127.0.0.1:6819/516459931] [v2:127.0.0.1:6820/516459931,v1:127.0.0.1:6821/516459931] exists,up 82af365f-20c6-4b19-8cb3-1af302acdb20 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:44:55.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:44:55.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:44:55.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:44:55.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:44:55.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2427a0b1-a687-46c1-9d18-21cf3e783c48 2024-04-25T08:44:55.646 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 2427a0b1-a687-46c1-9d18-21cf3e783c48 2024-04-25T08:44:55.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 2427a0b1-a687-46c1-9d18-21cf3e783c48' 2024-04-25T08:44:55.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:44:55.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHGCpmX/5lKBAA8vLrZ0naWeUNYo3lCabnog== 2024-04-25T08:44:55.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHGCpmX/5lKBAA8vLrZ0naWeUNYo3lCabnog=="}' 2024-04-25T08:44:55.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2427a0b1-a687-46c1-9d18-21cf3e783c48 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:44:56.204 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:44:56.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:44:56.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHGCpmX/5lKBAA8vLrZ0naWeUNYo3lCabnog== --osd-uuid 2427a0b1-a687-46c1-9d18-21cf3e783c48 2024-04-25T08:44:56.249 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:56.260+0000 7f58aab727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:56.317 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:56.328+0000 7f58aab727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:56.328+0000 7f58aab727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:44:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:56.328+0000 7f58aab727c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:44:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:56.328+0000 7f58aab727c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:44:56.319 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:56.328+0000 7f58aab727c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:44:56.424 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:44:56.432+0000 7f58aab727c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:45:03.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:45:03.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:45:03.768 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T08:45:03.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:45:03.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:45:04.354 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T08:45:04.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:45:04.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:45:04.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:45:04.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:45:04.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:45:04.387 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:04.395+0000 7f45bd4e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:04.389 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:04.399+0000 7f45bd4e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:04.390 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:04.399+0000 7f45bd4e37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:04.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:45:04.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:45:04.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:45:04.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:45:04.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:45:04.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:04.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:45:04.799 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:45:04.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:04.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:04.943 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:04.951+0000 7f45bd4e37c0 -1 Falling back to public interface 2024-04-25T08:45:05.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:06.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:06.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:06.244 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:45:06.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:45:06.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:06.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:06.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:07.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:07.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:07.654 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:45:07.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:45:07.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:07.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:07.670 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:07.679+0000 7f45bd4e37c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:45:08.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:09.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:09.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:09.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:45:09.070 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:45:09.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:09.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:09.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:10.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:10.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:10.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:45:10.847 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:45:10.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:10.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:11.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:12.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:12.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:12.409 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:45:12.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:45:12.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:12.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:13.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:14.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:14.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:14.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:45:14.072 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:45:14.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:14.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:14.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:15.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:15.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:15.744 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:45:15.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:45:15.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:15.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:16.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:17.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:17.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:17.202 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:45:17.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:45:17.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:17.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:17.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:18.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:18.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:18.748 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:45:18.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:45:18.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:18.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:19.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:20.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:20.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:20.322 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:45:20.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:45:20.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:20.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:20.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:21.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:21.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:21.897 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:45:21.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:45:21.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:21.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:22.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:23.412 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:45:23.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:45:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:23.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:24.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:25.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:25.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:25.464 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:45:25.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:45:25.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:25.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:26.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:27.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:27.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:27.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:45:27.384 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:45:27.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:27.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:28.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:29.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:29.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:29.303 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:45:29.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:45:29.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:29.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:30.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:31.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:31.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:31.151 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:45:31.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:45:31.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:31.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:31.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:32.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:32.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:32.884 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:45:32.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:45:32.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:32.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:33.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:34.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:34.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:34.981 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:45:34.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:45:34.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:34.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:35.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:36.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:36.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:36.613 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:45:36.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:45:36.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:36.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:45:37.100 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2809781057,v1:127.0.0.1:6827/2809781057] [v2:127.0.0.1:6828/2809781057,v1:127.0.0.1:6829/2809781057] exists,up 2427a0b1-a687-46c1-9d18-21cf3e783c48 2024-04-25T08:45:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:45:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:45:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:45:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-04-25T08:45:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:45:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:45:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:45:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:45:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:45:37.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:45:37.103 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:45:37.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:45:37.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:45:37.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:45:37.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:45:37.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:45:37.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:45:37.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:45:37.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:45:37.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:45:37.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:45:37.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:45:37.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:45:37.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:45:37.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:45:37.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:45:37.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c9c5bb1c-15a1-4ba6-81dc-eddd9a89d732 2024-04-25T08:45:37.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 c9c5bb1c-15a1-4ba6-81dc-eddd9a89d732' 2024-04-25T08:45:37.106 INFO:tasks.workunit.client.0.smithi102.stdout:add osd4 c9c5bb1c-15a1-4ba6-81dc-eddd9a89d732 2024-04-25T08:45:37.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:45:37.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxGCpmp24lCBAADLjldtu/BzFRBom1gwutQg== 2024-04-25T08:45:37.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxGCpmp24lCBAADLjldtu/BzFRBom1gwutQg=="}' 2024-04-25T08:45:37.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c9c5bb1c-15a1-4ba6-81dc-eddd9a89d732 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:45:37.626 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:45:37.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:45:37.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxGCpmp24lCBAADLjldtu/BzFRBom1gwutQg== --osd-uuid c9c5bb1c-15a1-4ba6-81dc-eddd9a89d732 2024-04-25T08:45:37.674 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:37.683+0000 7f0a7b1a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:37.712 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:37.719+0000 7f0a7b1a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:37.714 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:37.723+0000 7f0a7b1a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:37.714 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:37.723+0000 7f0a7b1a27c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:45:37.714 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:37.723+0000 7f0a7b1a27c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:45:37.714 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:37.723+0000 7f0a7b1a27c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:45:37.840 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:37.847+0000 7f0a7b1a27c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:45:44.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:45:44.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:45:44.859 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd4 key to auth repository 2024-04-25T08:45:44.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:45:44.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:45:45.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:45:45.421 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.4 2024-04-25T08:45:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:45:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:45:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:45:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:45:45.452 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:45.463+0000 7fa51b3787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:45.478 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:45.487+0000 7fa51b3787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:45.479 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:45.487+0000 7fa51b3787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:45.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:45:45.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:45:45.844 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:45:45.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:45:45.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:45:45.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:45:45.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:45.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:45:45.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:45.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:46.043 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:46.051+0000 7fa51b3787c0 -1 Falling back to public interface 2024-04-25T08:45:46.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:47.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:47.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:47.299 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:45:47.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:45:47.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:47.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:47.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:48.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:48.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:48.700 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:45:48.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:45:48.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:48.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:48.798 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:45:48.806+0000 7fa51b3787c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:45:49.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:50.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:50.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:50.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:45:50.180 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:45:50.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:50.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:50.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:51.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:51.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:51.652 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:45:51.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:45:51.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:51.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:52.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:53.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:53.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:53.262 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:45:53.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:45:53.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:53.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:53.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:54.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:54.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:54.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:45:54.922 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:45:54.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:54.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:55.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:56.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:56.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:56.365 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:45:56.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:45:56.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:56.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:56.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:57.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:57.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:57.935 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:45:57.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:45:57.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:57.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:58.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:59.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:59.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:59.420 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:45:59.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:45:59.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:59.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:45:59.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:00.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:00.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:00.907 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:46:00.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:46:00.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:00.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:01.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:02.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:02.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:02.381 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:46:02.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:46:02.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:02.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:02.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:03.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:03.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:03.923 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:46:03.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:46:03.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:03.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:04.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:05.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:05.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:05.819 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:46:05.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:46:05.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:05.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:06.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:07.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:07.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:07.935 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:46:07.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:46:07.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:07.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:08.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:09.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:09.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:09.732 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:46:09.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:46:09.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:09.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:10.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:11.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:11.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:11.577 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:46:11.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:46:11.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:11.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:12.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:13.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:13.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:13.195 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:46:13.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:46:13.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:13.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:14.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:15.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:15.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:15.032 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:46:15.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:46:15.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:15.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:46:15.495 INFO:tasks.workunit.client.0.smithi102.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2749452459,v1:127.0.0.1:6835/2749452459] [v2:127.0.0.1:6836/2749452459,v1:127.0.0.1:6837/2749452459] exists,up c9c5bb1c-15a1-4ba6-81dc-eddd9a89d732 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:46:15.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:46:15.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:46:15.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:46:15.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:46:15.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:46:15.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:46:15.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:46:15.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:46:15.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:46:15.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:46:15.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:46:15.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:46:15.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:46:15.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:46:15.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e26ee2f5-7048-4f9d-9032-d6ca951e6faa 2024-04-25T08:46:15.501 INFO:tasks.workunit.client.0.smithi102.stdout:add osd5 e26ee2f5-7048-4f9d-9032-d6ca951e6faa 2024-04-25T08:46:15.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 e26ee2f5-7048-4f9d-9032-d6ca951e6faa' 2024-04-25T08:46:15.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:46:15.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXGCpmhCjUHxAA2vfUzUIuD7ZaPaaN6fL/bA== 2024-04-25T08:46:15.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXGCpmhCjUHxAA2vfUzUIuD7ZaPaaN6fL/bA=="}' 2024-04-25T08:46:15.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e26ee2f5-7048-4f9d-9032-d6ca951e6faa -i td/osd-recovery-stats/5/new.json 2024-04-25T08:46:16.124 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:46:16.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:46:16.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXGCpmhCjUHxAA2vfUzUIuD7ZaPaaN6fL/bA== --osd-uuid e26ee2f5-7048-4f9d-9032-d6ca951e6faa 2024-04-25T08:46:16.171 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:16.178+0000 7f7df56ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:16.319 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:16.326+0000 7f7df56ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:16.320 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:16.330+0000 7f7df56ff7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:16.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:16.330+0000 7f7df56ff7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:46:16.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:16.330+0000 7f7df56ff7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:46:16.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:16.330+0000 7f7df56ff7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:46:16.459 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:16.466+0000 7f7df56ff7c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:46:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:46:23.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:46:23.375 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd5 key to auth repository 2024-04-25T08:46:23.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:46:23.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:46:23.923 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.5 2024-04-25T08:46:23.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:46:23.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:46:23.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:46:23.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:46:23.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:46:23.954 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:23.962+0000 7f02124637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:23.957 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:23.966+0000 7f02124637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:23.958 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:23.966+0000 7f02124637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:24.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:46:24.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:46:24.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:46:24.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:46:24.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:46:24.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:24.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:46:24.414 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:46:24.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:24.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:24.506 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:24.514+0000 7f02124637c0 -1 Falling back to public interface 2024-04-25T08:46:24.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:25.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:25.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:25.826 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:46:25.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:46:25.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:25.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:26.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:46:27.238+0000 7f02124637c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:46:27.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:27.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:27.257 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:46:27.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:46:27.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:27.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:29.084 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:46:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:46:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:29.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:30.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:30.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:30.692 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:46:30.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:46:30.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:30.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:31.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:32.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:32.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:32.139 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:46:32.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:46:32.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:32.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:32.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:33.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:33.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:33.651 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:46:33.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:46:33.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:33.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:34.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:35.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:35.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:35.194 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:46:35.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:46:35.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:35.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:35.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:36.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:36.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:36.634 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:46:36.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:46:36.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:36.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:37.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:38.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:38.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:38.165 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:46:38.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:46:38.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:38.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:38.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:39.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:39.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:39.639 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:46:39.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:46:39.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:39.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:40.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:41.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:41.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:41.152 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:46:41.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:46:41.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:41.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:41.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:42.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:42.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:42.636 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:46:42.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:46:42.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:42.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:43.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:44.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:44.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:44.438 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:46:44.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:46:44.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:44.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:45.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:46.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:46.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:46.321 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:46:46.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:46:46.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:46.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:46.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:47.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:47.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:47.821 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:46:47.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:46:47.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:47.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:48.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:49.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:49.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:49.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:46:49.649 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:46:49.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:49.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:50.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:51.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:51.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:51.433 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:46:51.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:46:51.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:51.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:52.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:53.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:53.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:53.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:46:53.998 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:46:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:53.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:46:54.479 INFO:tasks.workunit.client.0.smithi102.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1538195031,v1:127.0.0.1:6843/1538195031] [v2:127.0.0.1:6844/1538195031,v1:127.0.0.1:6845/1538195031] exists,up e26ee2f5-7048-4f9d-9032-d6ca951e6faa 2024-04-25T08:46:54.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:46:54.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:46:54.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:46:54.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' replicated = erasure ']' 2024-04-25T08:46:54.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:135: do_recovery_out1: create_pool test 1 1 replicated 2024-04-25T08:46:54.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 replicated 2024-04-25T08:46:55.124 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T08:46:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:46:56.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-04-25T08:46:56.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:46:56.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:46:56.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:46:56.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:46:56.145 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:46:56.145 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:46:56.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:46:56.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:46:56.145 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:46:56.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:46:56.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:46:56.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:46:56.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:46:56.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:46:56.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:46:56.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:46:56.681 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:46:56.681 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:46:56.681 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:46:56.681 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T08:46:56.681 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T08:46:56.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:46:56.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:56.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:46:56.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836526 2024-04-25T08:46:56.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836526 2024-04-25T08:46:56.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526' 2024-04-25T08:46:56.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:56.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:46:57.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673000 2024-04-25T08:46:57.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673000 2024-04-25T08:46:57.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673000' 2024-04-25T08:46:57.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:57.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:46:57.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509468 2024-04-25T08:46:57.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509468 2024-04-25T08:46:57.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509468' 2024-04-25T08:46:57.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:57.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:46:57.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345948 2024-04-25T08:46:57.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345948 2024-04-25T08:46:57.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509468 3-85899345948' 2024-04-25T08:46:57.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:57.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:46:57.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182417 2024-04-25T08:46:57.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182417 2024-04-25T08:46:57.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509468 3-85899345948 4-107374182417' 2024-04-25T08:46:57.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:46:57.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:46:57.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018892 2024-04-25T08:46:57.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018892 2024-04-25T08:46:57.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509468 3-85899345948 4-107374182417 5-128849018892' 2024-04-25T08:46:57.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:57.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836526 2024-04-25T08:46:57.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:57.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:46:57.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836526 2024-04-25T08:46:57.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:57.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836526 2024-04-25T08:46:57.722 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836526 2024-04-25T08:46:57.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-04-25T08:46:57.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:46:58.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836524 -lt 21474836526 2024-04-25T08:46:58.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:46:59.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:46:59.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:46:59.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836527 -lt 21474836526 2024-04-25T08:46:59.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:59.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673000 2024-04-25T08:46:59.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:59.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:46:59.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673000 2024-04-25T08:46:59.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:59.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673000 2024-04-25T08:46:59.538 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673000 2024-04-25T08:46:59.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2024-04-25T08:46:59.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:46:59.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673000 -lt 42949673000 2024-04-25T08:46:59.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:46:59.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509468 2024-04-25T08:46:59.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:46:59.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:46:59.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509468 2024-04-25T08:46:59.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:46:59.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509468 2024-04-25T08:46:59.970 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509468 2024-04-25T08:46:59.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-04-25T08:46:59.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:47:00.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509468 -lt 64424509468 2024-04-25T08:47:00.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:00.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345948 2024-04-25T08:47:00.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:00.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:47:00.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345948 2024-04-25T08:47:00.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:00.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345948 2024-04-25T08:47:00.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345948' 2024-04-25T08:47:00.409 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345948 2024-04-25T08:47:00.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:47:00.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345948 -lt 85899345948 2024-04-25T08:47:00.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:00.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182417 2024-04-25T08:47:00.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:00.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:47:00.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182417 2024-04-25T08:47:00.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:00.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182417 2024-04-25T08:47:00.835 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182417 2024-04-25T08:47:00.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2024-04-25T08:47:00.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:47:01.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182417 -lt 107374182417 2024-04-25T08:47:01.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:01.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018892 2024-04-25T08:47:01.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:01.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:47:01.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018892 2024-04-25T08:47:01.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:01.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018892 2024-04-25T08:47:01.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018892' 2024-04-25T08:47:01.253 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018892 2024-04-25T08:47:01.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:47:01.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018893 -lt 128849018892 2024-04-25T08:47:01.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:47:01.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:01.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:02.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:47:02.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:02.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:02.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:02.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:02.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:02.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:02.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:02.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:47:02.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:02.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:02.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:03.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:47:03.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:47:03.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:47:03.221 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-04-25T08:47:03.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:03.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-04-25T08:47:03.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:03.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-04-25T08:47:03.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:03.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-04-25T08:47:03.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:03.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-04-25T08:47:03.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:03.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-04-25T08:47:04.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:04.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-04-25T08:47:04.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:04.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-04-25T08:47:04.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:04.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-04-25T08:47:04.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:04.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-04-25T08:47:04.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:04.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-04-25T08:47:04.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:04.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-04-25T08:47:04.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:04.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-04-25T08:47:05.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:05.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-04-25T08:47:05.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:05.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-04-25T08:47:05.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:05.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-04-25T08:47:05.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:05.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-04-25T08:47:05.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:05.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-04-25T08:47:05.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:05.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-04-25T08:47:05.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:05.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-04-25T08:47:06.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:06.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-04-25T08:47:06.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:06.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-04-25T08:47:06.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:06.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-04-25T08:47:06.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:06.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-04-25T08:47:06.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:06.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-04-25T08:47:06.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:06.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-04-25T08:47:06.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:06.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-04-25T08:47:07.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:07.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-04-25T08:47:07.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:07.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-04-25T08:47:07.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:07.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-04-25T08:47:07.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:07.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-04-25T08:47:07.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:07.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-04-25T08:47:07.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:07.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-04-25T08:47:07.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:07.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-04-25T08:47:08.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:08.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-04-25T08:47:08.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:08.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-04-25T08:47:08.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:08.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-04-25T08:47:08.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:08.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-04-25T08:47:08.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:08.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-04-25T08:47:08.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:08.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-04-25T08:47:08.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:08.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-04-25T08:47:09.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:09.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-04-25T08:47:09.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:09.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-04-25T08:47:09.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:09.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-04-25T08:47:09.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:09.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-04-25T08:47:09.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:09.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-04-25T08:47:09.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:09.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-04-25T08:47:09.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:09.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-04-25T08:47:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-04-25T08:47:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-04-25T08:47:10.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:10.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-04-25T08:47:10.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:10.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-04-25T08:47:10.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:10.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-04-25T08:47:10.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:10.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-04-25T08:47:10.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:10.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-04-25T08:47:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-04-25T08:47:11.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:11.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-04-25T08:47:11.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:11.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-04-25T08:47:11.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:11.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-04-25T08:47:11.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:11.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-04-25T08:47:11.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:11.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-04-25T08:47:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:11.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-04-25T08:47:12.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:12.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-04-25T08:47:12.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:12.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-04-25T08:47:12.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:12.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-04-25T08:47:12.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:12.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-04-25T08:47:12.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:12.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-04-25T08:47:12.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:12.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-04-25T08:47:12.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:12.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-04-25T08:47:13.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:13.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-04-25T08:47:13.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:13.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-04-25T08:47:13.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:13.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-04-25T08:47:13.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:13.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-04-25T08:47:13.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:13.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-04-25T08:47:13.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:13.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-04-25T08:47:13.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:13.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-04-25T08:47:14.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:14.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-04-25T08:47:14.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:14.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-04-25T08:47:14.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:14.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-04-25T08:47:14.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:14.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-04-25T08:47:14.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:14.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-04-25T08:47:14.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:14.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-04-25T08:47:14.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:14.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-04-25T08:47:14.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:14.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-04-25T08:47:15.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:15.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-04-25T08:47:15.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:15.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-04-25T08:47:15.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:15.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-04-25T08:47:15.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:15.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-04-25T08:47:15.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:15.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-04-25T08:47:15.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:15.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-04-25T08:47:15.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:15.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-04-25T08:47:16.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:16.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-04-25T08:47:16.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:16.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-04-25T08:47:16.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:16.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-04-25T08:47:16.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:16.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-04-25T08:47:16.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:16.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-04-25T08:47:16.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:16.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-04-25T08:47:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:17.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-04-25T08:47:17.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:17.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-04-25T08:47:17.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:17.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-04-25T08:47:17.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:17.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-04-25T08:47:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:17.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-04-25T08:47:17.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:17.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-04-25T08:47:17.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:17.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-04-25T08:47:18.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:18.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-04-25T08:47:18.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:18.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-04-25T08:47:18.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:18.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-04-25T08:47:18.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:18.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-04-25T08:47:18.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:18.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-04-25T08:47:18.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:18.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-04-25T08:47:18.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:18.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-04-25T08:47:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:19.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-04-25T08:47:19.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:19.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-04-25T08:47:19.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:19.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-04-25T08:47:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-04-25T08:47:19.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:19.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-04-25T08:47:19.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:19.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-04-25T08:47:19.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:19.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-04-25T08:47:20.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:20.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-04-25T08:47:20.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:20.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-04-25T08:47:20.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:20.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-04-25T08:47:20.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:20.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-04-25T08:47:20.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:20.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-04-25T08:47:20.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:20.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-04-25T08:47:20.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:20.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-04-25T08:47:21.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:21.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-04-25T08:47:21.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:21.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-04-25T08:47:21.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:21.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-04-25T08:47:21.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:21.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-04-25T08:47:21.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:21.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-04-25T08:47:21.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:21.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-04-25T08:47:21.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:21.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-04-25T08:47:22.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:22.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-04-25T08:47:22.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:22.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-04-25T08:47:22.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:22.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-04-25T08:47:22.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:22.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-04-25T08:47:22.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:22.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-04-25T08:47:22.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:22.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-04-25T08:47:22.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:22.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-04-25T08:47:23.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:23.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-04-25T08:47:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:23.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-04-25T08:47:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-04-25T08:47:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-04-25T08:47:23.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:23.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-04-25T08:47:24.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:24.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-04-25T08:47:24.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:24.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-04-25T08:47:24.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:24.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-04-25T08:47:24.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:24.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-04-25T08:47:24.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:24.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-04-25T08:47:25.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:25.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-04-25T08:47:25.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:25.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-04-25T08:47:25.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:25.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-04-25T08:47:25.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:25.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-04-25T08:47:25.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:25.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-04-25T08:47:25.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:25.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-04-25T08:47:25.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:25.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-04-25T08:47:26.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:26.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-04-25T08:47:26.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:26.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-04-25T08:47:26.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:26.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-04-25T08:47:26.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:26.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-04-25T08:47:26.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:26.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-04-25T08:47:26.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:26.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-04-25T08:47:27.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:27.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-04-25T08:47:27.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:27.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-04-25T08:47:27.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:27.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-04-25T08:47:27.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:27.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-04-25T08:47:27.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:27.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-04-25T08:47:27.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:27.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-04-25T08:47:27.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:27.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-04-25T08:47:27.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:27.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-04-25T08:47:28.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:28.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-04-25T08:47:28.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:28.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-04-25T08:47:28.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:28.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-04-25T08:47:28.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:28.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-04-25T08:47:28.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:28.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-04-25T08:47:28.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:28.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-04-25T08:47:28.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:28.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-04-25T08:47:29.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:29.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-04-25T08:47:29.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:29.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-04-25T08:47:29.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:29.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-04-25T08:47:29.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:29.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-04-25T08:47:29.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:29.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-04-25T08:47:29.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:29.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-04-25T08:47:29.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:29.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-04-25T08:47:30.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:30.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-04-25T08:47:30.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:30.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-04-25T08:47:30.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:30.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-04-25T08:47:30.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:30.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-04-25T08:47:30.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:30.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-04-25T08:47:30.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:30.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-04-25T08:47:30.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:30.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-04-25T08:47:31.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:31.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-04-25T08:47:31.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:31.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-04-25T08:47:31.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:31.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-04-25T08:47:31.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:31.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-04-25T08:47:31.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:31.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-04-25T08:47:31.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:31.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-04-25T08:47:31.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:31.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-04-25T08:47:32.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:32.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-04-25T08:47:32.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:32.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-04-25T08:47:32.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:47:32.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-04-25T08:47:32.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-04-25T08:47:32.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:47:32.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:47:32.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:47:32.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:47:32.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-04-25T08:47:32.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-04-25T08:47:32.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:47:32.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:47:32.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:47:32.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:47:33.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-04-25T08:47:33.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-04-25T08:47:33.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:47:33.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:47:33.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:47:33.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:47:33.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:47:33.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:47:33.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:47:33.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:47:33.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:47:33.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:47:34.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-04-25T08:47:34.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-04-25T08:47:35.121 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is set 2024-04-25T08:47:35.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:47:35.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 327313 2024-04-25T08:47:35.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-04-25T08:47:35.912 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T08:47:35.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-04-25T08:47:36.598 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. 2024-04-25T08:47:36.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-04-25T08:47:37.542 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is unset 2024-04-25T08:47:37.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-04-25T08:47:37.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:47:37.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:47:37.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:47:37.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:47:38.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:47:38.263 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:47:38.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-04-25T08:47:40.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-04-25T08:47:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:47:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:47:40.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:47:40.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:47:40.283 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:47:40.283 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:47:40.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:47:40.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:47:40.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:47:40.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:47:40.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:47:40.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:47:40.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:47:40.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:47:40.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:47:40.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:47:40.909 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:47:40.909 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:47:40.909 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:47:40.909 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T08:47:40.909 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T08:47:40.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:47:40.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:40.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:47:41.110 INFO:tasks.workunit.client.0.smithi102.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:47:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:47:41.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:47:41.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:47:41.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:41.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:47:41.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673010 2024-04-25T08:47:41.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673010 2024-04-25T08:47:41.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673010' 2024-04-25T08:47:41.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:41.269 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:47:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509479 2024-04-25T08:47:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509479 2024-04-25T08:47:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673010 2-64424509479' 2024-04-25T08:47:41.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:41.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:47:41.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345959 2024-04-25T08:47:41.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345959 2024-04-25T08:47:41.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673010 2-64424509479 3-85899345959' 2024-04-25T08:47:41.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:41.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:47:41.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182428 2024-04-25T08:47:41.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182428 2024-04-25T08:47:41.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673010 2-64424509479 3-85899345959 4-107374182428' 2024-04-25T08:47:41.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:47:41.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:47:41.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018903 2024-04-25T08:47:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018903 2024-04-25T08:47:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673010 2-64424509479 3-85899345959 4-107374182428 5-128849018903' 2024-04-25T08:47:41.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:41.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673010 2024-04-25T08:47:41.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:41.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:47:41.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673010 2024-04-25T08:47:41.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:41.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673010 2024-04-25T08:47:41.885 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673010 2024-04-25T08:47:41.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2024-04-25T08:47:41.886 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:47:42.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673008 -lt 42949673010 2024-04-25T08:47:42.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:47:43.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:47:43.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:47:43.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673011 -lt 42949673010 2024-04-25T08:47:43.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:43.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509479 2024-04-25T08:47:43.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:43.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:47:43.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509479 2024-04-25T08:47:43.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:43.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509479 2024-04-25T08:47:43.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-04-25T08:47:43.731 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509479 2024-04-25T08:47:43.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:47:44.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509479 -lt 64424509479 2024-04-25T08:47:44.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:44.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345959 2024-04-25T08:47:44.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:44.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:47:44.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345959 2024-04-25T08:47:44.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:44.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345959 2024-04-25T08:47:44.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345959' 2024-04-25T08:47:44.274 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345959 2024-04-25T08:47:44.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:47:44.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345959 -lt 85899345959 2024-04-25T08:47:44.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:44.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182428 2024-04-25T08:47:44.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:44.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:47:44.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182428 2024-04-25T08:47:44.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:44.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182428 2024-04-25T08:47:44.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182428' 2024-04-25T08:47:44.848 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182428 2024-04-25T08:47:44.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:47:45.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182428 -lt 107374182428 2024-04-25T08:47:45.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:47:45.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018903 2024-04-25T08:47:45.301 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:47:45.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:47:45.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:47:45.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018903 2024-04-25T08:47:45.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018903 2024-04-25T08:47:45.304 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018903 2024-04-25T08:47:45.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018903' 2024-04-25T08:47:45.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:47:45.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018903 -lt 128849018903 2024-04-25T08:47:45.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:47:45.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:45.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:46.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:47:46.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:46.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:46.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:46.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:46.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:46.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:46.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:46.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:46.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:46.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:46.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:47.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:47.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:47:47.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:47:47.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:47:47.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:47.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:47.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:47.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:47.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:47.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:47.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:47.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:47.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:47.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:48.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:48.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:48.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:48.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:48.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:48.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:48.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:48.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:48.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:48.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:48.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:48.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:48.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:49.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:47:49.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:47:49.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:49.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:49.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:49.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:49.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:49.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:49.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:49.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:49.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:49.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:49.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:49.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:49.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:49.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:49.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:50.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:50.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:50.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:50.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:50.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:50.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:50.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:50.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:50.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:51.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:47:51.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:47:51.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:51.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:51.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:51.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:51.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:51.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:51.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:51.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:51.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:51.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:51.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:51.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:51.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:51.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:51.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:52.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:52.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:52.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:52.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:52.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:52.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:52.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:52.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:52.302 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:52.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13 2024-04-25T08:47:52.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13 '!=' null 2024-04-25T08:47:52.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:52.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:52.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:53.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:53.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:53.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:53.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:53.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:53.003 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:53.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:53.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:53.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:53.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:53.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:53.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:53.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:53.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:53.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:53.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:53.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:53.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:53.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:53.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:53.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:54.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13 2024-04-25T08:47:54.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13 '!=' null 2024-04-25T08:47:54.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:54.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:54.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:54.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:54.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:54.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:54.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:54.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:54.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:54.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:54.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:55.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:55.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:55.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:55.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:55.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:55.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:55.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:55.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:55.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:55.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:55.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:55.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:55.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:56.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13 2024-04-25T08:47:56.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13 '!=' null 2024-04-25T08:47:56.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:56.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:56.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:56.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:56.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:56.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:56.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:56.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:56.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:56.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:56.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:56.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:47:56.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:56.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:56.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:57.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:47:57.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:47:57.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:47:57.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:47:57.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:47:57.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:47:57.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:47:57.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:47:57.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:47:57.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:47:57.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:47:57.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:47:57.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:47:57.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:47:58.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:47:58.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:47:58.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:47:58.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:47:58.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:47:58.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:47:58.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:47:58.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:47:58.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:47:58.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:47:58.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:47:58.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:47:58.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:47:58.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:47:58.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:47:58.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 replicated 200 0 0 0 2024-04-25T08:47:58.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:47:58.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:47:58.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:47:58.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:47:58.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:47:59.000 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:47:59.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2,4]/[1,2]' 2024-04-25T08:47:59.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:47:59.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:47:59.011 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:47:59.011 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2,4]/[1,2] ' 2024-04-25T08:47:59.012 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:47:59.012 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:47:59.012 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:47:59.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:47:59.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:47:59.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:47:59.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:47:59.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:47:59.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:47:59.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:47:59.018 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:47:59.019 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:47:59.019 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:47:59.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:47:59.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:47:59.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:47:59.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:47:59.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:47:59.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:47:59.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:47:59.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:47:59.027 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:47:59.028 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2,4]/[1,2] ' 2024-04-25T08:47:59.028 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:47:59.028 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:47:59.028 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:47:59.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T08:47:59.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T08:47:59.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:47:59.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:47:59.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:47:59.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:47:59.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:47:59.034 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:47:59.034 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:47:59.034 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:47:59.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:47:59.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:47:59.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:47:59.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:47:59.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:47:59.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:47:59.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:47:59.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-04-25T08:47:59.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:47:59.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:47:59.587 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T08:47:59.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-04-25T08:47:59.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:47:59.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:47:59.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:47:59.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:47:59.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:48:04.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:48:04.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:48:04.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:48:04.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:48:04.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:48:04.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:48:04.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:48:04.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:48:04.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:48:04.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:48:04.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:48:04.933 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:48:04.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:48:04.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:48:04.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:48:04.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:48:04.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:48:04.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:48:04.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:48:04.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:48:04.937 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:48:04.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:48:04.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:48:04.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:48:04.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:48:04.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:04.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:48:04.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:48:04.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:48:04.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:48:04.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:48:04.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:48:04.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:48:04.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:48:04.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:48:04.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:48:04.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:48:04.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:48:04.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:48:04.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:48:04.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:48:04.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:48:04.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:48:04.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:48:04.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:48:04.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:48:04.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:48:04.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:48:04.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:48:04.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:48:04.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:48:04.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:48:04.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:48:04.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:48:04.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:48:04.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:48:04.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:48:04.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:04.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:48:04.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:48:04.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:48:04.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:48:04.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:48:04.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:48:04.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:04.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:48:04.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.277516 2024-04-25T08:48:04.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizedown td/osd-recovery-stats 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:241: TEST_recovery_sizedown: local dir=td/osd-recovery-stats 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:243: TEST_recovery_sizedown: run_mon td/osd-recovery-stats a 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:48:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:48:05.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:48:05.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:48:05.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:48:05.472 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:48:05.472 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:05.472 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:48:05.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:48:05.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:48:05.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:48:05.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:48:05.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:48:05.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:48:05.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:48:05.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:48:05.703 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:48:05.703 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:48:05.703 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:48:05.703 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:48:05.703 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:05.703 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:48:05.704 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:48:05.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:48:05.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get fsid 2024-04-25T08:48:05.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:48:05.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:48:05.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:48:05.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:48:05.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:48:05.822 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:48:05.822 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:48:05.822 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:48:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:48:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:48:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:48:05.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:48:05.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get mon_host 2024-04-25T08:48:05.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:244: TEST_recovery_sizedown: run_mgr td/osd-recovery-stats x 2024-04-25T08:48:05.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:48:05.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:48:05.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:48:05.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:48:05.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:48:05.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:48:06.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:48:06.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:48:06.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:48:06.846 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:48:06.846 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:06.846 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:48:06.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:48:06.846 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:48:06.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:48:06.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:245: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 0 2024-04-25T08:48:06.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:48:06.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:48:06.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:48:06.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:48:06.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:48:06.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:48:06.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:48:06.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:48:06.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:48:06.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:48:06.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:48:06.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=51dd3102-7aac-465b-a789-dccbba42cefe 2024-04-25T08:48:06.890 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 51dd3102-7aac-465b-a789-dccbba42cefe 2024-04-25T08:48:06.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 51dd3102-7aac-465b-a789-dccbba42cefe' 2024-04-25T08:48:06.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:48:06.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDGGCpmnuWUNhAAyX1b9HK4VmjwXE1+CNLXwg== 2024-04-25T08:48:06.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDGGCpmnuWUNhAAyX1b9HK4VmjwXE1+CNLXwg=="}' 2024-04-25T08:48:06.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 51dd3102-7aac-465b-a789-dccbba42cefe -i td/osd-recovery-stats/0/new.json 2024-04-25T08:48:07.218 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:48:07.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:48:07.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDGGCpmnuWUNhAAyX1b9HK4VmjwXE1+CNLXwg== --osd-uuid 51dd3102-7aac-465b-a789-dccbba42cefe 2024-04-25T08:48:07.264 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:07.268+0000 7f5667d5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:07.295 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:07.300+0000 7f5667d5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:07.296 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:07.304+0000 7f5667d5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:07.297 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:07.304+0000 7f5667d5e7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:48:07.297 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:07.304+0000 7f5667d5e7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:48:07.297 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:07.304+0000 7f5667d5e7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:48:07.335 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:07.340+0000 7f5667d5e7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:48:13.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:48:13.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:48:13.683 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:48:13.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:48:13.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:48:14.228 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:48:14.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:48:14.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:48:14.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:48:14.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:48:14.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:48:14.261 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:14.268+0000 7fa8a94347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:14.263 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:14.268+0000 7fa8a94347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:14.265 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:14.272+0000 7fa8a94347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:14.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:48:14.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:48:14.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:48:14.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:48:14.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:48:14.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:14.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:48:14.618 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:48:14.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:14.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:14.813 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:14.820+0000 7fa8a94347c0 -1 Falling back to public interface 2024-04-25T08:48:15.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:16.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:16.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:16.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:48:16.070 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:48:16.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:16.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:16.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:17.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:17.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:17.476 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:48:17.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:48:17.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:17.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:17.512 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:17.524+0000 7fa8a94347c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:48:18.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:19.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:19.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:19.082 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:48:19.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:48:19.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:19.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:19.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:20.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:20.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:20.650 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:48:20.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:48:20.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:20.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:21.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:22.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:22.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:22.140 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:48:22.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:48:22.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:22.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:22.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:23.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:23.703 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:48:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:48:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:24.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:25.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:25.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:25.185 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:48:25.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:48:25.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:25.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:25.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:26.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:26.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:26.748 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:48:26.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:48:26.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:26.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:27.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:28.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:28.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:28.247 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:48:28.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:48:28.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:28.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:28.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:29.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:29.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:29.734 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:48:29.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:48:29.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:29.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:30.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:31.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:31.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:31.188 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:48:31.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:48:31.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:31.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:31.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:32.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:32.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:32.931 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:48:32.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:48:32.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:32.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:33.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:34.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:34.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:34.623 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:48:34.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:48:34.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:34.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:35.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:36.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:36.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:36.243 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:48:36.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:48:36.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:36.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:37.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:38.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:38.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:38.393 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:48:38.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:48:38.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:38.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:39.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:40.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:40.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:40.629 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:48:40.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:48:40.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:40.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:41.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:42.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:48:42.376 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:48:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:42.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:43.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:44.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:44.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:44.126 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:48:44.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:48:44.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:44.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:48:44.552 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3337235533,v1:127.0.0.1:6803/3337235533] [v2:127.0.0.1:6804/3337235533,v1:127.0.0.1:6805/3337235533] exists,up 51dd3102-7aac-465b-a789-dccbba42cefe 2024-04-25T08:48:44.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:48:44.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:48:44.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:48:44.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:246: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 1 2024-04-25T08:48:44.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:48:44.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:48:44.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:48:44.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:48:44.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:48:44.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:48:44.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:48:44.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:48:44.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:48:44.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:48:44.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e1e5001-b904-4360-b320-ef52ab1be053 2024-04-25T08:48:44.561 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 0e1e5001-b904-4360-b320-ef52ab1be053 2024-04-25T08:48:44.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0e1e5001-b904-4360-b320-ef52ab1be053' 2024-04-25T08:48:44.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:48:44.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsGCpmPYnjIhAAsOuetaowOngqPzSOXNaeEQ== 2024-04-25T08:48:44.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsGCpmPYnjIhAAsOuetaowOngqPzSOXNaeEQ=="}' 2024-04-25T08:48:44.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e1e5001-b904-4360-b320-ef52ab1be053 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:48:45.134 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:48:45.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:48:45.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsGCpmPYnjIhAAsOuetaowOngqPzSOXNaeEQ== --osd-uuid 0e1e5001-b904-4360-b320-ef52ab1be053 2024-04-25T08:48:45.181 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:45.191+0000 7fb04f4057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:45.211 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:45.219+0000 7fb04f4057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:45.213 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:45.223+0000 7fb04f4057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:45.214 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:45.223+0000 7fb04f4057c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:48:45.214 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:45.223+0000 7fb04f4057c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:48:45.214 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:45.223+0000 7fb04f4057c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:48:45.242 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:45.251+0000 7fb04f4057c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:48:52.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:48:52.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:48:52.254 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T08:48:52.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:48:52.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:48:52.830 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:48:52.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:48:52.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:48:52.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:48:52.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:48:52.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:48:52.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:52.871+0000 7fd5b40477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:52.901 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:52.911+0000 7fd5b40477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:52.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:52.911+0000 7fd5b40477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:53.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:48:53.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:48:53.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:48:53.279 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:48:53.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:48:53.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:48:53.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:53.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:48:53.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:53.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:53.481 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:53.491+0000 7fd5b40477c0 -1 Falling back to public interface 2024-04-25T08:48:53.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:54.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:54.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:54.726 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:48:54.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:48:54.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:54.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:55.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:56.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:56.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:56.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:48:56.132 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:48:56.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:56.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:56.188 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:48:56.199+0000 7fd5b40477c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:48:56.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:57.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:57.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:57.590 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:48:57.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:48:57.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:57.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:58.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:59.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:59.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:59.009 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:48:59.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:48:59.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:59.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:48:59.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:00.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:00.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:00.480 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:49:00.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:49:00.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:00.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:00.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:01.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:01.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:01.919 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:49:01.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:49:01.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:01.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:02.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:03.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:03.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:03.511 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:49:03.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:49:03.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:03.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:03.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:04.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:04.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:04.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:49:04.923 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:49:04.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:04.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:05.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:06.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:06.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:06.520 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:49:06.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:49:06.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:06.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:06.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:07.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:07.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:07.960 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:49:07.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:49:07.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:07.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:08.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:09.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:09.471 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:49:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:49:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:09.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:10.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:10.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:10.937 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:49:10.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:49:10.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:10.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:11.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:12.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:12.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:12.769 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:49:12.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:49:12.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:12.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:13.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:14.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:14.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:14.476 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:49:14.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:49:14.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:14.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:15.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:16.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:16.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:16.315 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:49:16.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:49:16.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:16.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:17.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:18.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:18.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:18.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:49:18.224 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:49:18.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:18.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:19.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:20.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:20.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:20.069 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:49:20.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:49:20.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:20.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:20.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:21.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:49:21.963 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:49:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:21.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:22.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:23.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:23.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:23.567 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:49:23.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:49:23.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:23.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:49:23.903 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2427198659,v1:127.0.0.1:6811/2427198659] [v2:127.0.0.1:6812/2427198659,v1:127.0.0.1:6813/2427198659] exists,up 0e1e5001-b904-4360-b320-ef52ab1be053 2024-04-25T08:49:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:49:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:49:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:49:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:247: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 2 2024-04-25T08:49:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:49:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:49:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:49:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:49:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:49:23.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:49:23.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:49:23.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:49:23.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:49:23.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:49:23.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:49:23.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:49:23.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:49:23.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:49:23.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:49:23.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:49:23.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:49:23.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:49:23.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:49:23.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:49:23.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:49:23.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:49:23.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=955fee22-4b7a-4716-9e89-1892a3be0552 2024-04-25T08:49:23.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 955fee22-4b7a-4716-9e89-1892a3be0552' 2024-04-25T08:49:23.911 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 955fee22-4b7a-4716-9e89-1892a3be0552 2024-04-25T08:49:23.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:49:23.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATGSpm6QUnOBAAr/vQXq4PFybUfoDWCjMvgA== 2024-04-25T08:49:23.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATGSpm6QUnOBAAr/vQXq4PFybUfoDWCjMvgA=="}' 2024-04-25T08:49:23.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 955fee22-4b7a-4716-9e89-1892a3be0552 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:49:24.437 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:49:24.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:49:24.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATGSpm6QUnOBAAr/vQXq4PFybUfoDWCjMvgA== --osd-uuid 955fee22-4b7a-4716-9e89-1892a3be0552 2024-04-25T08:49:24.482 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:24.490+0000 7f787247c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:24.524 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:24.534+0000 7f787247c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:24.526 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:24.534+0000 7f787247c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:24.527 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:24.534+0000 7f787247c7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:49:24.527 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:24.534+0000 7f787247c7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:49:24.527 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:24.534+0000 7f787247c7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:49:24.572 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:24.582+0000 7f787247c7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:49:31.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:49:31.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:49:31.796 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T08:49:31.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:49:31.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:49:32.348 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T08:49:32.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:49:32.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:49:32.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:49:32.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:49:32.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:49:32.381 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:32.390+0000 7f41a911c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:32.383 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:32.390+0000 7f41a911c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:32.384 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:32.394+0000 7f41a911c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:32.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:49:32.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:49:32.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:49:32.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:49:32.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:49:32.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:32.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:49:32.765 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:49:32.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:32.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:32.945 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:32.954+0000 7f41a911c7c0 -1 Falling back to public interface 2024-04-25T08:49:33.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:34.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:34.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:34.215 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:49:34.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:49:34.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:34.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:34.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:35.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:35.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:35.643 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:49:35.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:49:35.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:35.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:35.654 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:49:35.662+0000 7f41a911c7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:49:36.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:37.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:37.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:37.168 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:49:37.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:49:37.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:37.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:37.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:38.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:38.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:38.773 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:49:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:49:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:38.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:39.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:40.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:40.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:40.198 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:49:40.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:49:40.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:40.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:40.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:41.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:41.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:41.764 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:49:41.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:49:41.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:41.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:42.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:43.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:43.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:43.178 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:49:43.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:49:43.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:43.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:43.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:44.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:44.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:44.831 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:49:44.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:49:44.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:44.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:45.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:46.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:46.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:46.249 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:49:46.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:49:46.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:46.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:46.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:47.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:47.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:47.765 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:49:47.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:49:47.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:47.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:48.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:49.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:49.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:49.334 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:49:49.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:49:49.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:49.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:49.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:50.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:50.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:50.950 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:49:50.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:49:50.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:50.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:51.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:52.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:52.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:52.379 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:49:52.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:49:52.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:52.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:53.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:54.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:54.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:54.312 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:49:54.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:49:54.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:54.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:55.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:56.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:56.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:56.146 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:49:56.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:49:56.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:56.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:57.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:58.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:58.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:58.110 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:49:58.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:49:58.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:58.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:49:59.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:00.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:00.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:00.088 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:50:00.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:50:00.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:00.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:50:01.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:02.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:50:02.169 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:50:02.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:02.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:50:02.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:03.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:03.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:03.676 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:50:03.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:50:03.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:03.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:50:04.117 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1178254870,v1:127.0.0.1:6819/1178254870] [v2:127.0.0.1:6820/1178254870,v1:127.0.0.1:6821/1178254870] exists,up 955fee22-4b7a-4716-9e89-1892a3be0552 2024-04-25T08:50:04.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:50:04.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:50:04.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:50:04.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:248: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 3 2024-04-25T08:50:04.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:50:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:50:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:50:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:50:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:50:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:50:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:50:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:50:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:50:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:50:04.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:50:04.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:50:04.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:50:04.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:50:04.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:50:04.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:50:04.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:50:04.127 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:50:04.127 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:50:04.127 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:50:04.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:50:04.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:50:04.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:50:04.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:50:04.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:50:04.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:50:04.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:50:04.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:50:04.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:50:04.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:50:04.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:50:04.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:50:04.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:50:04.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:50:04.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:50:04.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4463dcf-2fe4-4373-8d5c-87add20b7931 2024-04-25T08:50:04.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b4463dcf-2fe4-4373-8d5c-87add20b7931' 2024-04-25T08:50:04.129 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 b4463dcf-2fe4-4373-8d5c-87add20b7931 2024-04-25T08:50:04.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:50:04.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8GSpmziMACRAAvfMT7vkPCZig9VVArIw6zQ== 2024-04-25T08:50:04.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8GSpmziMACRAAvfMT7vkPCZig9VVArIw6zQ=="}' 2024-04-25T08:50:04.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4463dcf-2fe4-4373-8d5c-87add20b7931 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:50:04.657 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:50:04.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:50:04.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8GSpmziMACRAAvfMT7vkPCZig9VVArIw6zQ== --osd-uuid b4463dcf-2fe4-4373-8d5c-87add20b7931 2024-04-25T08:50:04.707 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:04.713+0000 7f3e274fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:04.864 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:04.873+0000 7f3e274fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:04.866 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:04.873+0000 7f3e274fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:04.867 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:04.873+0000 7f3e274fb7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:50:04.867 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:04.873+0000 7f3e274fb7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:50:04.867 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:04.873+0000 7f3e274fb7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:50:04.911 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:04.917+0000 7f3e274fb7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:50:10.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:50:10.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:50:10.626 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T08:50:10.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:50:10.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:50:11.186 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T08:50:11.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:50:11.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:50:11.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:50:11.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:50:11.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:50:11.219 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:11.225+0000 7fea98f887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:11.221 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:11.229+0000 7fea98f887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:11.222 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:11.229+0000 7fea98f887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:11.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:50:11.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:50:11.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:50:11.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:50:11.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:50:11.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:11.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:50:11.649 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:50:11.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:11.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:11.780 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:11.789+0000 7fea98f887c0 -1 Falling back to public interface 2024-04-25T08:50:12.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:13.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:13.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:13.108 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:50:13.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:50:13.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:13.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:13.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:14.329 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:14.337+0000 7fea98f887c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:50:14.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:14.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:14.510 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:50:14.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:50:14.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:14.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:15.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:16.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:16.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:16.178 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:50:16.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:50:16.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:16.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:16.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:17.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:17.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:17.598 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:50:17.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:50:17.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:17.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:18.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:19.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:19.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:19.037 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:50:19.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:50:19.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:19.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:19.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:20.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:20.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:20.563 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:50:20.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:50:20.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:20.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:21.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:22.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:22.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:22.009 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:50:22.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:50:22.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:22.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:22.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:23.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:23.476 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:50:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:50:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:23.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:23.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:24.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:24.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:24.915 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:50:24.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:50:24.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:24.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:25.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:26.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:26.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:26.389 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:50:26.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:50:26.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:26.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:26.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:27.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:27.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:27.821 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:50:27.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:50:27.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:27.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:28.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:29.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:29.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:29.375 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:50:29.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:50:29.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:29.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:30.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:31.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:31.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:31.126 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:50:31.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:50:31.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:31.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:31.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:32.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:32.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:32.893 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:50:32.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:50:32.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:32.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:33.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:34.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:34.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:34.861 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:50:34.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:50:34.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:34.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:35.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:36.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:36.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:36.764 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:50:36.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:50:36.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:36.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:38.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:39.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:39.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:39.140 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:50:39.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:50:39.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:39.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:39.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:40.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:40.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:40.958 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:50:40.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:50:40.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:40.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:50:41.474 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1985904935,v1:127.0.0.1:6827/1985904935] [v2:127.0.0.1:6828/1985904935,v1:127.0.0.1:6829/1985904935] exists,up b4463dcf-2fe4-4373-8d5c-87add20b7931 2024-04-25T08:50:41.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:50:41.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:50:41.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:50:41.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:249: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 4 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:50:41.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:50:41.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:50:41.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:50:41.478 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:50:41.478 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:50:41.478 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:50:41.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:50:41.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:50:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:50:41.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:50:41.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=50ea4e34-db01-4e71-a765-3d63084c47d0 2024-04-25T08:50:41.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 50ea4e34-db01-4e71-a765-3d63084c47d0' 2024-04-25T08:50:41.481 INFO:tasks.workunit.client.0.smithi102.stdout:add osd4 50ea4e34-db01-4e71-a765-3d63084c47d0 2024-04-25T08:50:41.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:50:41.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhGSpm2cq7HhAADmXEA8D5lw8JOcvvaFLIeg== 2024-04-25T08:50:41.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhGSpm2cq7HhAADmXEA8D5lw8JOcvvaFLIeg=="}' 2024-04-25T08:50:41.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 50ea4e34-db01-4e71-a765-3d63084c47d0 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:50:42.007 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:50:42.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:50:42.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhGSpm2cq7HhAADmXEA8D5lw8JOcvvaFLIeg== --osd-uuid 50ea4e34-db01-4e71-a765-3d63084c47d0 2024-04-25T08:50:42.065 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:42.073+0000 7f50db60c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:42.119 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:42.125+0000 7f50db60c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:42.120 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:42.129+0000 7f50db60c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:42.129+0000 7f50db60c7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:50:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:42.129+0000 7f50db60c7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:50:42.121 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:42.129+0000 7f50db60c7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:50:42.174 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:42.181+0000 7f50db60c7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:50:47.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:50:47.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:50:47.964 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd4 key to auth repository 2024-04-25T08:50:47.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:50:47.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:50:48.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:50:48.516 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.4 2024-04-25T08:50:48.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:50:48.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:50:48.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:50:48.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:50:48.546 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:48.553+0000 7f13f77c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:48.566 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:48.573+0000 7f13f77c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:48.567 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:48.577+0000 7f13f77c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:48.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:50:48.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:50:48.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:50:48.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:50:48.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:50:48.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:48.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:50:48.917 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:50:48.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:48.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:49.126 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:49.133+0000 7f13f77c37c0 -1 Falling back to public interface 2024-04-25T08:50:49.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:50.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:50.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:50.364 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:50:50.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:50:50.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:50.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:50.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:51.675 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:50:51.681+0000 7f13f77c37c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:50:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:51.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:51.794 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:50:51.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:50:51.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:51.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:52.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:53.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:53.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:53.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:50:53.396 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:50:53.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:53.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:53.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:54.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:54.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:54.887 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:50:54.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:50:54.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:54.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:55.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:56.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:56.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:56.299 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:50:56.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:50:56.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:56.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:56.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:57.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:50:57.727 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:50:57.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:57.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:58.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:59.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:59.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:59.289 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:50:59.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:50:59.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:59.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:50:59.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:00.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:00.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:00.795 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:51:00.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:51:00.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:00.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:51:01.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:02.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:02.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:02.246 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:51:02.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:51:02.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:02.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:51:02.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:03.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:03.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:03.751 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:51:03.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:51:03.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:03.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:51:04.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:05.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:05.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:05.205 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:51:05.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:51:05.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:05.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:51:05.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:06.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:06.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:06.941 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:51:06.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:51:06.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:06.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:51:07.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:08.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:08.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:08.852 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:51:08.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:51:08.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:08.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:51:09.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:10.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:10.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:10.691 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:51:10.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:51:10.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:10.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:51:11.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:12.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:12.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:12.471 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:51:12.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:51:12.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:12.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:51:12.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:13.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:13.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:13.962 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:51:13.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:51:13.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:13.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:51:14.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:15.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:15.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:15.440 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:51:15.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:51:15.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:15.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:51:15.887 INFO:tasks.workunit.client.0.smithi102.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2338803493,v1:127.0.0.1:6835/2338803493] [v2:127.0.0.1:6836/2338803493,v1:127.0.0.1:6837/2338803493] exists,up 50ea4e34-db01-4e71-a765-3d63084c47d0 2024-04-25T08:51:15.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:51:15.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:51:15.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:51:15.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:250: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 5 2024-04-25T08:51:15.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:51:15.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:51:15.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:51:15.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:51:15.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:51:15.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:15.892 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:51:15.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:51:15.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a0a261a0-ea91-41cb-bb27-4adf416678b5 2024-04-25T08:51:15.894 INFO:tasks.workunit.client.0.smithi102.stdout:add osd5 a0a261a0-ea91-41cb-bb27-4adf416678b5 2024-04-25T08:51:15.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 a0a261a0-ea91-41cb-bb27-4adf416678b5' 2024-04-25T08:51:15.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:51:15.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDGSpmATU3NxAAU1A2wuXOK7IClKAsTbNQsw== 2024-04-25T08:51:15.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDGSpmATU3NxAAU1A2wuXOK7IClKAsTbNQsw=="}' 2024-04-25T08:51:15.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a0a261a0-ea91-41cb-bb27-4adf416678b5 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:51:16.367 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:51:16.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:51:16.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCDGSpmATU3NxAAU1A2wuXOK7IClKAsTbNQsw== --osd-uuid a0a261a0-ea91-41cb-bb27-4adf416678b5 2024-04-25T08:51:16.419 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:16.424+0000 7f2f879897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:16.537 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:16.544+0000 7f2f879897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:16.539 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:16.548+0000 7f2f879897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:16.540 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:16.548+0000 7f2f879897c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:51:16.540 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:16.548+0000 7f2f879897c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:51:16.540 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:16.548+0000 7f2f879897c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:51:16.667 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:16.676+0000 7f2f879897c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:51:23.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:51:23.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:51:23.057 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd5 key to auth repository 2024-04-25T08:51:23.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:51:23.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:51:23.604 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.5 2024-04-25T08:51:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:51:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:51:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:51:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:51:23.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:51:23.636 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:23.644+0000 7eff8105f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:23.655 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:23.664+0000 7eff8105f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:23.656 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:23.664+0000 7eff8105f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:24.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:51:24.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:51:24.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:51:24.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:51:24.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:51:24.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:24.082 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:51:24.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:51:24.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:24.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:24.200 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:24.208+0000 7eff8105f7c0 -1 Falling back to public interface 2024-04-25T08:51:24.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:25.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:25.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:25.524 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:51:25.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:51:25.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:25.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:25.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:26.625 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:51:26.632+0000 7eff8105f7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:51:26.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:26.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:26.944 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:51:26.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:51:26.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:26.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:27.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:28.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:28.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:28.460 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:51:28.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:51:28.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:28.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:28.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:29.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:29.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:29.925 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:51:29.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:51:29.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:29.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:30.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:31.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:31.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:31.449 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:51:31.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:51:31.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:31.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:32.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:33.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:33.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:33.064 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:51:33.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:51:33.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:33.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:33.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:34.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:34.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:34.588 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:51:34.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:51:34.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:34.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:35.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:36.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:36.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:36.019 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:51:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:51:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:36.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:36.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:37.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:37.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:37.675 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:51:37.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:51:37.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:37.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:38.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:39.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:39.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:39.080 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:51:39.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:51:39.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:39.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:39.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:40.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:40.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:40.590 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:51:40.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:51:40.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:40.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:41.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:42.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:42.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:42.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:51:42.752 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:51:42.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:42.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:43.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:44.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:44.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:44.275 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:51:44.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:51:44.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:44.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:44.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:45.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:45.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:45.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:51:45.937 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:51:45.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:45.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:46.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:47.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:47.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:47.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:51:47.660 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:51:47.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:47.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:48.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:49.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:49.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:49.313 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:51:49.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:51:49.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:49.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:50.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:51.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:51.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:51.225 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:51:51.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:51:51.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:51.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:51:51.771 INFO:tasks.workunit.client.0.smithi102.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3188023512,v1:127.0.0.1:6843/3188023512] [v2:127.0.0.1:6844/3188023512,v1:127.0.0.1:6845/3188023512] exists,up a0a261a0-ea91-41cb-bb27-4adf416678b5 2024-04-25T08:51:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:51:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:51:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:51:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:252: TEST_recovery_sizedown: create_pool test 1 1 2024-04-25T08:51:51.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:51:52.367 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T08:51:52.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:51:53.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:253: TEST_recovery_sizedown: ceph osd pool set test size 4 2024-04-25T08:51:54.648 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 4 2024-04-25T08:51:54.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:255: TEST_recovery_sizedown: wait_for_clean 2024-04-25T08:51:54.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:51:54.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:51:54.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:51:54.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:51:54.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:51:54.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:51:54.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:51:54.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:51:54.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:51:54.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:51:54.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:51:54.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:51:54.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:51:54.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:51:54.775 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:51:55.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:51:55.403 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:51:55.403 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:51:55.403 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:51:55.404 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T08:51:55.404 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T08:51:55.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:51:55.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:55.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:51:55.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836523 2024-04-25T08:51:55.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836523 2024-04-25T08:51:55.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523' 2024-04-25T08:51:55.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:55.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:51:55.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672998 2024-04-25T08:51:55.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672998 2024-04-25T08:51:55.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523 1-42949672998' 2024-04-25T08:51:55.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:55.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:51:55.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509470 2024-04-25T08:51:55.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509470 2024-04-25T08:51:55.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523 1-42949672998 2-64424509470' 2024-04-25T08:51:55.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:55.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:51:56.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345941 2024-04-25T08:51:56.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345941 2024-04-25T08:51:56.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523 1-42949672998 2-64424509470 3-85899345941' 2024-04-25T08:51:56.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:56.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:51:56.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182416 2024-04-25T08:51:56.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182416 2024-04-25T08:51:56.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523 1-42949672998 2-64424509470 3-85899345941 4-107374182416' 2024-04-25T08:51:56.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:51:56.340 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:51:56.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018888 2024-04-25T08:51:56.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018888 2024-04-25T08:51:56.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836523 1-42949672998 2-64424509470 3-85899345941 4-107374182416 5-128849018888' 2024-04-25T08:51:56.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:56.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836523 2024-04-25T08:51:56.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:56.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:51:56.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836523 2024-04-25T08:51:56.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:56.493 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836523 2024-04-25T08:51:56.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836523 2024-04-25T08:51:56.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-04-25T08:51:56.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:51:56.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836521 -lt 21474836523 2024-04-25T08:51:56.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:51:57.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:51:57.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:51:58.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836523 -lt 21474836523 2024-04-25T08:51:58.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:58.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:58.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672998 2024-04-25T08:51:58.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:51:58.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672998 2024-04-25T08:51:58.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:58.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672998 2024-04-25T08:51:58.386 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672998 2024-04-25T08:51:58.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-04-25T08:51:58.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:51:58.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672998 -lt 42949672998 2024-04-25T08:51:58.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:58.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:58.819 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509470 2024-04-25T08:51:58.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:51:58.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509470 2024-04-25T08:51:58.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:58.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509470 2024-04-25T08:51:58.822 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509470 2024-04-25T08:51:58.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-04-25T08:51:58.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:51:59.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509470 -lt 64424509470 2024-04-25T08:51:59.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:59.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345941 2024-04-25T08:51:59.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:59.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:51:59.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345941 2024-04-25T08:51:59.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:59.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345941 2024-04-25T08:51:59.245 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345941 2024-04-25T08:51:59.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-04-25T08:51:59.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:51:59.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345941 -lt 85899345941 2024-04-25T08:51:59.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:51:59.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182416 2024-04-25T08:51:59.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:51:59.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:51:59.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182416 2024-04-25T08:51:59.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:51:59.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182416 2024-04-25T08:51:59.687 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182416 2024-04-25T08:51:59.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182416' 2024-04-25T08:51:59.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:52:00.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182416 -lt 107374182416 2024-04-25T08:52:00.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:00.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018888 2024-04-25T08:52:00.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:00.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:52:00.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018888 2024-04-25T08:52:00.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:00.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018888 2024-04-25T08:52:00.101 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018888 2024-04-25T08:52:00.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2024-04-25T08:52:00.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:52:00.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018889 -lt 128849018888 2024-04-25T08:52:00.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:52:00.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:00.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:01.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:52:01.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:01.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:01.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:01.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:01.065 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:01.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:01.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:01.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:52:01.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:01.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:01.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:02.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:52:02.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:52:02.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:52:02.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: seq 1 200 2024-04-25T08:52:02.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:02.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj1 /dev/null 2024-04-25T08:52:02.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:02.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj2 /dev/null 2024-04-25T08:52:02.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:02.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj3 /dev/null 2024-04-25T08:52:02.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:02.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj4 /dev/null 2024-04-25T08:52:02.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:02.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj5 /dev/null 2024-04-25T08:52:03.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:03.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj6 /dev/null 2024-04-25T08:52:03.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:03.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj7 /dev/null 2024-04-25T08:52:03.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:03.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj8 /dev/null 2024-04-25T08:52:03.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:03.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj9 /dev/null 2024-04-25T08:52:03.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:03.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj10 /dev/null 2024-04-25T08:52:03.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:03.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj11 /dev/null 2024-04-25T08:52:03.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:03.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj12 /dev/null 2024-04-25T08:52:04.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:04.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj13 /dev/null 2024-04-25T08:52:04.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:04.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj14 /dev/null 2024-04-25T08:52:04.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:04.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj15 /dev/null 2024-04-25T08:52:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj16 /dev/null 2024-04-25T08:52:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:04.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj17 /dev/null 2024-04-25T08:52:04.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:04.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj18 /dev/null 2024-04-25T08:52:05.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:05.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj19 /dev/null 2024-04-25T08:52:05.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:05.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj20 /dev/null 2024-04-25T08:52:05.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:05.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj21 /dev/null 2024-04-25T08:52:05.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:05.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj22 /dev/null 2024-04-25T08:52:05.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:05.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj23 /dev/null 2024-04-25T08:52:05.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:05.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj24 /dev/null 2024-04-25T08:52:05.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:05.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj25 /dev/null 2024-04-25T08:52:06.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:06.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj26 /dev/null 2024-04-25T08:52:06.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:06.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj27 /dev/null 2024-04-25T08:52:06.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:06.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj28 /dev/null 2024-04-25T08:52:06.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:06.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj29 /dev/null 2024-04-25T08:52:06.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:06.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj30 /dev/null 2024-04-25T08:52:06.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:06.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj31 /dev/null 2024-04-25T08:52:07.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:07.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj32 /dev/null 2024-04-25T08:52:07.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:07.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj33 /dev/null 2024-04-25T08:52:07.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:07.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj34 /dev/null 2024-04-25T08:52:07.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:07.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj35 /dev/null 2024-04-25T08:52:07.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:07.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj36 /dev/null 2024-04-25T08:52:07.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:07.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj37 /dev/null 2024-04-25T08:52:07.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:07.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj38 /dev/null 2024-04-25T08:52:08.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:08.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj39 /dev/null 2024-04-25T08:52:08.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:08.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj40 /dev/null 2024-04-25T08:52:08.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:08.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj41 /dev/null 2024-04-25T08:52:08.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:08.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj42 /dev/null 2024-04-25T08:52:08.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:08.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj43 /dev/null 2024-04-25T08:52:08.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:08.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj44 /dev/null 2024-04-25T08:52:09.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:09.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj45 /dev/null 2024-04-25T08:52:09.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:09.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj46 /dev/null 2024-04-25T08:52:09.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:09.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj47 /dev/null 2024-04-25T08:52:09.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:09.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj48 /dev/null 2024-04-25T08:52:09.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:09.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj49 /dev/null 2024-04-25T08:52:09.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:09.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj50 /dev/null 2024-04-25T08:52:10.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:10.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj51 /dev/null 2024-04-25T08:52:10.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:10.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj52 /dev/null 2024-04-25T08:52:10.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:10.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj53 /dev/null 2024-04-25T08:52:10.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:10.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj54 /dev/null 2024-04-25T08:52:10.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:10.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj55 /dev/null 2024-04-25T08:52:10.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:10.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj56 /dev/null 2024-04-25T08:52:11.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:11.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj57 /dev/null 2024-04-25T08:52:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:11.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj58 /dev/null 2024-04-25T08:52:11.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:11.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj59 /dev/null 2024-04-25T08:52:11.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:11.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj60 /dev/null 2024-04-25T08:52:11.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:11.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj61 /dev/null 2024-04-25T08:52:11.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:11.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj62 /dev/null 2024-04-25T08:52:11.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:11.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj63 /dev/null 2024-04-25T08:52:12.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:12.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj64 /dev/null 2024-04-25T08:52:12.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:12.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj65 /dev/null 2024-04-25T08:52:12.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:12.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj66 /dev/null 2024-04-25T08:52:12.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:12.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj67 /dev/null 2024-04-25T08:52:12.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:12.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj68 /dev/null 2024-04-25T08:52:12.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:12.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj69 /dev/null 2024-04-25T08:52:13.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:13.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj70 /dev/null 2024-04-25T08:52:13.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:13.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj71 /dev/null 2024-04-25T08:52:13.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:13.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj72 /dev/null 2024-04-25T08:52:13.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:13.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj73 /dev/null 2024-04-25T08:52:13.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:13.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj74 /dev/null 2024-04-25T08:52:13.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:13.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj75 /dev/null 2024-04-25T08:52:13.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:13.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj76 /dev/null 2024-04-25T08:52:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj77 /dev/null 2024-04-25T08:52:14.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:14.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj78 /dev/null 2024-04-25T08:52:14.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:14.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj79 /dev/null 2024-04-25T08:52:14.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:14.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj80 /dev/null 2024-04-25T08:52:14.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:14.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj81 /dev/null 2024-04-25T08:52:14.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:14.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj82 /dev/null 2024-04-25T08:52:15.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:15.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj83 /dev/null 2024-04-25T08:52:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:15.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj84 /dev/null 2024-04-25T08:52:15.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:15.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj85 /dev/null 2024-04-25T08:52:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:15.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj86 /dev/null 2024-04-25T08:52:15.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:15.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj87 /dev/null 2024-04-25T08:52:15.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:15.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj88 /dev/null 2024-04-25T08:52:16.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:16.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj89 /dev/null 2024-04-25T08:52:16.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:16.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj90 /dev/null 2024-04-25T08:52:16.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:16.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj91 /dev/null 2024-04-25T08:52:16.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:16.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj92 /dev/null 2024-04-25T08:52:16.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:16.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj93 /dev/null 2024-04-25T08:52:16.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:16.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj94 /dev/null 2024-04-25T08:52:16.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:16.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj95 /dev/null 2024-04-25T08:52:17.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:17.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj96 /dev/null 2024-04-25T08:52:17.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:17.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj97 /dev/null 2024-04-25T08:52:17.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:17.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj98 /dev/null 2024-04-25T08:52:17.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:17.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj99 /dev/null 2024-04-25T08:52:17.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:17.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj100 /dev/null 2024-04-25T08:52:17.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:17.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj101 /dev/null 2024-04-25T08:52:18.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:18.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj102 /dev/null 2024-04-25T08:52:18.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:18.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj103 /dev/null 2024-04-25T08:52:18.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:18.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj104 /dev/null 2024-04-25T08:52:18.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:18.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj105 /dev/null 2024-04-25T08:52:18.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:18.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj106 /dev/null 2024-04-25T08:52:18.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:18.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj107 /dev/null 2024-04-25T08:52:19.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:19.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj108 /dev/null 2024-04-25T08:52:19.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:19.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj109 /dev/null 2024-04-25T08:52:19.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:19.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj110 /dev/null 2024-04-25T08:52:19.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:19.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj111 /dev/null 2024-04-25T08:52:19.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:19.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj112 /dev/null 2024-04-25T08:52:19.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:19.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj113 /dev/null 2024-04-25T08:52:19.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:19.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj114 /dev/null 2024-04-25T08:52:20.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:20.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj115 /dev/null 2024-04-25T08:52:20.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:20.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj116 /dev/null 2024-04-25T08:52:20.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:20.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj117 /dev/null 2024-04-25T08:52:20.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:20.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj118 /dev/null 2024-04-25T08:52:20.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:20.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj119 /dev/null 2024-04-25T08:52:20.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:20.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj120 /dev/null 2024-04-25T08:52:21.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:21.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj121 /dev/null 2024-04-25T08:52:21.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:21.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj122 /dev/null 2024-04-25T08:52:21.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:21.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj123 /dev/null 2024-04-25T08:52:21.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:21.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj124 /dev/null 2024-04-25T08:52:21.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:21.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj125 /dev/null 2024-04-25T08:52:21.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:21.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj126 /dev/null 2024-04-25T08:52:21.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:21.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj127 /dev/null 2024-04-25T08:52:22.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:22.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj128 /dev/null 2024-04-25T08:52:22.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:22.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj129 /dev/null 2024-04-25T08:52:22.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:22.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj130 /dev/null 2024-04-25T08:52:22.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:22.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj131 /dev/null 2024-04-25T08:52:22.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:22.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj132 /dev/null 2024-04-25T08:52:22.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:22.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj133 /dev/null 2024-04-25T08:52:23.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:23.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj134 /dev/null 2024-04-25T08:52:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:23.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj135 /dev/null 2024-04-25T08:52:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:23.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj136 /dev/null 2024-04-25T08:52:23.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:23.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj137 /dev/null 2024-04-25T08:52:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:23.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj138 /dev/null 2024-04-25T08:52:23.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:23.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj139 /dev/null 2024-04-25T08:52:24.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:24.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj140 /dev/null 2024-04-25T08:52:24.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:24.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj141 /dev/null 2024-04-25T08:52:24.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:24.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj142 /dev/null 2024-04-25T08:52:24.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:24.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj143 /dev/null 2024-04-25T08:52:24.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:24.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj144 /dev/null 2024-04-25T08:52:24.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:24.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj145 /dev/null 2024-04-25T08:52:24.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:24.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj146 /dev/null 2024-04-25T08:52:25.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:25.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj147 /dev/null 2024-04-25T08:52:25.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:25.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj148 /dev/null 2024-04-25T08:52:25.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:25.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj149 /dev/null 2024-04-25T08:52:25.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:25.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj150 /dev/null 2024-04-25T08:52:25.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:25.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj151 /dev/null 2024-04-25T08:52:25.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:25.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj152 /dev/null 2024-04-25T08:52:26.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:26.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj153 /dev/null 2024-04-25T08:52:26.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:26.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj154 /dev/null 2024-04-25T08:52:26.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:26.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj155 /dev/null 2024-04-25T08:52:26.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:26.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj156 /dev/null 2024-04-25T08:52:26.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:26.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj157 /dev/null 2024-04-25T08:52:26.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:26.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj158 /dev/null 2024-04-25T08:52:27.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:27.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj159 /dev/null 2024-04-25T08:52:27.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:27.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj160 /dev/null 2024-04-25T08:52:27.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:27.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj161 /dev/null 2024-04-25T08:52:27.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:27.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj162 /dev/null 2024-04-25T08:52:27.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:27.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj163 /dev/null 2024-04-25T08:52:27.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:27.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj164 /dev/null 2024-04-25T08:52:27.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:27.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj165 /dev/null 2024-04-25T08:52:28.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:28.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj166 /dev/null 2024-04-25T08:52:28.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:28.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj167 /dev/null 2024-04-25T08:52:28.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:28.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj168 /dev/null 2024-04-25T08:52:28.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:28.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj169 /dev/null 2024-04-25T08:52:28.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:28.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj170 /dev/null 2024-04-25T08:52:28.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:28.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj171 /dev/null 2024-04-25T08:52:29.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:29.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj172 /dev/null 2024-04-25T08:52:29.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:29.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj173 /dev/null 2024-04-25T08:52:29.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:29.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj174 /dev/null 2024-04-25T08:52:29.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:29.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj175 /dev/null 2024-04-25T08:52:29.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:29.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj176 /dev/null 2024-04-25T08:52:29.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:29.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj177 /dev/null 2024-04-25T08:52:29.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:29.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj178 /dev/null 2024-04-25T08:52:30.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:30.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj179 /dev/null 2024-04-25T08:52:30.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:30.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj180 /dev/null 2024-04-25T08:52:30.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:30.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj181 /dev/null 2024-04-25T08:52:30.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:30.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj182 /dev/null 2024-04-25T08:52:30.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:30.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj183 /dev/null 2024-04-25T08:52:30.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:30.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj184 /dev/null 2024-04-25T08:52:31.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:31.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj185 /dev/null 2024-04-25T08:52:31.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:31.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj186 /dev/null 2024-04-25T08:52:31.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:31.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj187 /dev/null 2024-04-25T08:52:31.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:31.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj188 /dev/null 2024-04-25T08:52:31.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:31.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj189 /dev/null 2024-04-25T08:52:31.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:31.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj190 /dev/null 2024-04-25T08:52:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:32.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj191 /dev/null 2024-04-25T08:52:32.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:32.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj192 /dev/null 2024-04-25T08:52:32.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:32.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj193 /dev/null 2024-04-25T08:52:32.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:32.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj194 /dev/null 2024-04-25T08:52:32.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:32.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj195 /dev/null 2024-04-25T08:52:32.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:32.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj196 /dev/null 2024-04-25T08:52:32.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:32.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj197 /dev/null 2024-04-25T08:52:33.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:33.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj198 /dev/null 2024-04-25T08:52:33.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:33.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj199 /dev/null 2024-04-25T08:52:33.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T08:52:33.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj200 /dev/null 2024-04-25T08:52:33.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T08:52:33.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:52:33.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:52:33.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:52:33.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:52:34.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: local primary=1 2024-04-25T08:52:34.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: get_pg test obj1 2024-04-25T08:52:34.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:52:34.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:52:34.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:52:34.146 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:52:34.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: local PG=1.0 2024-04-25T08:52:34.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: get_osds test obj1 2024-04-25T08:52:34.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-04-25T08:52:34.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj1 2024-04-25T08:52:34.625 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test obj1 2024-04-25T08:52:34.625 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T08:52:35.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-25T08:52:35.077 INFO:tasks.workunit.client.0.smithi102.stderr:0 2024-04-25T08:52:35.077 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:52:35.077 INFO:tasks.workunit.client.0.smithi102.stderr:4' 2024-04-25T08:52:35.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 4 2024-04-25T08:52:35.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: local 'allosds=1 0 2 4' 2024-04-25T08:52:35.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:267: TEST_recovery_sizedown: ceph osd set norecover 2024-04-25T08:52:35.767 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is set 2024-04-25T08:52:35.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:52:35.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.1 2024-04-25T08:52:36.414 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already out. 2024-04-25T08:52:36.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:52:36.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.0 2024-04-25T08:52:37.057 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. 2024-04-25T08:52:37.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:52:37.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.2 2024-04-25T08:52:38.039 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already out. 2024-04-25T08:52:38.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T08:52:38.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.4 2024-04-25T08:52:39.259 INFO:tasks.workunit.client.0.smithi102.stderr:osd.4 is already out. 2024-04-25T08:52:39.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:273: TEST_recovery_sizedown: ceph osd pool set test size 2 2024-04-25T08:52:40.892 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-04-25T08:52:40.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:274: TEST_recovery_sizedown: ceph osd unset norecover 2024-04-25T08:52:42.124 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is unset 2024-04-25T08:52:42.150 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T08:52:42.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:52:42.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:52:42.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:52:42.152 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:52:42.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: ceph tell osd.3 debug kick_recovery_wq 0 2024-04-25T08:52:42.826 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:52:42.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:276: TEST_recovery_sizedown: sleep 2 2024-04-25T08:52:44.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:278: TEST_recovery_sizedown: wait_for_clean 2024-04-25T08:52:44.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:52:44.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:52:44.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:52:44.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:52:44.848 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:52:44.848 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:52:44.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:52:44.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:52:44.849 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:52:44.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:52:44.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:52:44.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:52:44.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:52:44.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:52:44.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:52:45.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:52:45.396 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:52:45.396 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:52:45.396 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:52:45.396 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T08:52:45.396 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T08:52:45.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:52:45.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:45.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:52:45.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836535 2024-04-25T08:52:45.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836535 2024-04-25T08:52:45.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836535' 2024-04-25T08:52:45.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:45.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:52:45.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673010 2024-04-25T08:52:45.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673010 2024-04-25T08:52:45.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836535 1-42949673010' 2024-04-25T08:52:45.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:45.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:52:45.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509482 2024-04-25T08:52:45.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509482 2024-04-25T08:52:45.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836535 1-42949673010 2-64424509482' 2024-04-25T08:52:45.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:45.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:52:46.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345953 2024-04-25T08:52:46.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345953 2024-04-25T08:52:46.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836535 1-42949673010 2-64424509482 3-85899345953' 2024-04-25T08:52:46.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:46.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:52:46.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182428 2024-04-25T08:52:46.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182428 2024-04-25T08:52:46.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836535 1-42949673010 2-64424509482 3-85899345953 4-107374182428' 2024-04-25T08:52:46.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:52:46.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:52:46.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018900 2024-04-25T08:52:46.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018900 2024-04-25T08:52:46.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836535 1-42949673010 2-64424509482 3-85899345953 4-107374182428 5-128849018900' 2024-04-25T08:52:46.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:46.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:46.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836535 2024-04-25T08:52:46.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:52:46.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836535 2024-04-25T08:52:46.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:46.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836535 2024-04-25T08:52:46.447 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836535 2024-04-25T08:52:46.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-04-25T08:52:46.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:52:46.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836533 -lt 21474836535 2024-04-25T08:52:46.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:52:47.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:52:47.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:52:48.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836535 -lt 21474836535 2024-04-25T08:52:48.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:48.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673010 2024-04-25T08:52:48.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:48.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:52:48.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673010 2024-04-25T08:52:48.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:48.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673010 2024-04-25T08:52:48.286 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673010 2024-04-25T08:52:48.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2024-04-25T08:52:48.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:52:48.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673010 -lt 42949673010 2024-04-25T08:52:48.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:48.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509482 2024-04-25T08:52:48.764 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:48.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:52:48.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509482 2024-04-25T08:52:48.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:48.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509482 2024-04-25T08:52:48.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-04-25T08:52:48.768 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509482 2024-04-25T08:52:48.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:52:49.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509482 -lt 64424509482 2024-04-25T08:52:49.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:49.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345953 2024-04-25T08:52:49.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:49.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:52:49.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:49.208 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345953 2024-04-25T08:52:49.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345953 2024-04-25T08:52:49.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345953' 2024-04-25T08:52:49.209 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345953 2024-04-25T08:52:49.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:52:49.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345953 -lt 85899345953 2024-04-25T08:52:49.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:49.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182428 2024-04-25T08:52:49.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:49.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:52:49.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:49.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182428 2024-04-25T08:52:49.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182428 2024-04-25T08:52:49.704 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182428 2024-04-25T08:52:49.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182428' 2024-04-25T08:52:49.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:52:50.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182428 -lt 107374182428 2024-04-25T08:52:50.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:52:50.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018900 2024-04-25T08:52:50.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:52:50.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:52:50.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018900 2024-04-25T08:52:50.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:52:50.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018900 2024-04-25T08:52:50.109 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018900 2024-04-25T08:52:50.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018900' 2024-04-25T08:52:50.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:52:50.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018901 -lt 128849018900 2024-04-25T08:52:50.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:52:50.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:50.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:51.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:52:51.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:51.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:51.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:51.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:51.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:51.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:51.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:51.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:51.582 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:51.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:51.583 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:52.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:52.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:52:52.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:52:52.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:52:52.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:52.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:52.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:52.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:52.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:52.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:52.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:52.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:52.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:52.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:52.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:52.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:52.701 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:52.701 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:53.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:53.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:53.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:53.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:53.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:53.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:53.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:53.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:53.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:53.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:52:53.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:52:53.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:53.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:53.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:53.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:53.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:53.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:53.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:53.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:53.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:53.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:53.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:54.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:54.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:54.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:54.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:54.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:54.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:55.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:52:55.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:52:55.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:55.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:55.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:55.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:55.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:55.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:55.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:55.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:55.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:55.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:55.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:56.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:56.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:56.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:56.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:56.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:56.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:56.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:56.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:56.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:56.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:56.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:56.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:56.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:57.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:52:57.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:52:57.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:57.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:57.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:57.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:57.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:57.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:57.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:57.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:57.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:57.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:57.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:58.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:58.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:58.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:58.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:52:58.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:52:58.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:52:58.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:52:58.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:52:58.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:52:58.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:52:58.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:52:58.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:52:58.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:52:59.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:52:59.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:52:59.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:52:59.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:52:59.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:52:59.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:52:59.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:52:59.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:52:59.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:52:59.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:52:59.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:52:59.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:52:59.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:52:59.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:52:59.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:52:59.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:52:59.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:53:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:53:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:53:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:53:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:53:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:53:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:53:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:53:00.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:53:00.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:53:01.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:53:01.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:53:01.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:53:01.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:53:01.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:53:01.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:53:01.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:53:01.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:53:01.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:53:01.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:53:01.289 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:53:01.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:53:01.290 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:53:01.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:53:01.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:53:01.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:53:01.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:53:02.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:53:02.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:53:02.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:53:02.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:53:02.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:53:02.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:53:02.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:53:02.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:53:02.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:53:02.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-04-25T08:53:02.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-04-25T08:53:02.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:53:02.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:53:02.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:53:02.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:53:02.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:53:02.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:53:02.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:53:02.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:53:02.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:53:02.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:53:02.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:53:03.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:53:03.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:53:03.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:53:03.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:53:03.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:53:03.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:53:03.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:53:03.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:53:03.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:53:03.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:53:03.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:53:03.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:53:03.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:53:04.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:53:04.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:53:04.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:53:04.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:53:04.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:53:04.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:53:04.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:53:04.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:53:04.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:53:04.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:53:04.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:53:04.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:53:04.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:53:05.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:53:05.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:53:05.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:53:05.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:53:05.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:53:05.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:53:05.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:53:05.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:53:05.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:53:05.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:53:05.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:53:05.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:53:05.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:53:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:53:06.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:53:06.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:53:06.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:53:06.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:53:06.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:53:06.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:53:06.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:53:06.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:53:06.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:53:06.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:53:06.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:53:06.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:53:07.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:53:07.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:53:07.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:53:07.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:53:07.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:53:07.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:53:07.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:53:07.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:53:07.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:53:07.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:53:07.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:53:07.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:53:07.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:53:08.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-04-25T08:53:08.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-04-25T08:53:08.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:53:08.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:53:08.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:53:08.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:53:08.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:53:08.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:53:08.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:53:08.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:53:08.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:53:08.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:53:08.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:53:08.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:53:08.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:53:08.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:53:08.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:53:09.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:53:09.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:53:09.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:53:09.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:53:09.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:53:09.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:53:09.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:53:09.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:53:09.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:53:09.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:53:09.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:53:09.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:53:09.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:53:09.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:53:09.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:53:09.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:53:09.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:53:09.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:53:09.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:53:09.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:53:09.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:53:09.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:53:10.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:53:10.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:53:10.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:53:10.341 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:53:10.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:53:10.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:53:10.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:53:10.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:53:10.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:53:10.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:53:10.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:53:10.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:53:10.851 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:53:11.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:53:11.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:53:11.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:53:11.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:53:11.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:53:11.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:53:11.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:53:11.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:53:11.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:53:11.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:53:11.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:53:11.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:53:11.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:53:11.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:53:11.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:53:11.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:53:11.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:53:12.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:53:12.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:53:12.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:53:12.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T08:53:12.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:53:12.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:53:12.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:53:12.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:53:12.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: primary=3 2024-04-25T08:53:12.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: expr 200 '*' 2 2024-04-25T08:53:12.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: local misplaced=400 2024-04-25T08:53:12.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:284: TEST_recovery_sizedown: local log=td/osd-recovery-stats/osd.3.log 2024-04-25T08:53:12.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:285: TEST_recovery_sizedown: check td/osd-recovery-stats 1.0 3 replicated 0 0 400 0 2024-04-25T08:53:12.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:53:12.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:53:12.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=3 2024-04-25T08:53:12.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:53:12.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=0 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.3.log 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:53:12.895 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:53:12.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[3,5]/[3]' 2024-04-25T08:53:12.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:53:12.909 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:53:12.910 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:53:12.910 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [3,5]/[3] ' 2024-04-25T08:53:12.910 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:53:12.910 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:53:12.910 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:53:12.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=0 2024-04-25T08:53:12.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 0 0 2024-04-25T08:53:12.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:53:12.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:53:12.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:53:12.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:53:12.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:53:12.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:53:12.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:53:12.915 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:53:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:53:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:53:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:53:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:53:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:53:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:53:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:53:12.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:53:12.923 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:53:12.923 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [3,5]/[3] ' 2024-04-25T08:53:12.923 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:53:12.923 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:53:12.923 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:53:12.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=399 2024-04-25T08:53:12.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 399 400 2024-04-25T08:53:12.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=399 2024-04-25T08:53:12.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:53:12.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T08:53:12.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:53:12.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:53:12.927 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:53:12.927 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:53:12.927 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:53:12.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:53:12.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:53:12.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:53:12.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:53:12.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:53:12.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:53:12.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:53:12.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:53:12.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: tail -1 2024-04-25T08:53:12.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:53:12.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: UPACT='[3,5]/[3]' 2024-04-25T08:53:12.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:53:12.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep -F ' [3,5]/[3] ' 2024-04-25T08:53:12.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: head -1 2024-04-25T08:53:12.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:53:12.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: FIRST=200 2024-04-25T08:53:12.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:291: TEST_recovery_sizedown: below_margin 200 200 2024-04-25T08:53:12.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:53:12.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:53:12.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:53:12.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:53:12.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-04-25T08:53:12.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: tail -1 2024-04-25T08:53:12.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:53:12.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: LAST=0 2024-04-25T08:53:12.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:293: TEST_recovery_sizedown: above_margin 0 0 2024-04-25T08:53:12.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:53:12.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:53:12.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:53:12.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:53:12.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:295: TEST_recovery_sizedown: delete_pool test 2024-04-25T08:53:12.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:53:12.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:53:13.479 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T08:53:13.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:296: TEST_recovery_sizedown: kill_daemons td/osd-recovery-stats 2024-04-25T08:53:13.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:53:13.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:53:13.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:53:13.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:53:13.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:53:14.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:53:14.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:53:14.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:53:14.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:53:14.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:53:14.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:53:14.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:53:14.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:53:14.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:53:14.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:53:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:53:14.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:53:14.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:53:14.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:53:14.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:53:14.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:53:14.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:53:14.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:53:14.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:53:14.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:53:14.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:53:14.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:53:14.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:53:14.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:53:14.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:53:14.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:14.869 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:53:14.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:53:14.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:53:14.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:53:14.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:53:14.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:53:14.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:53:14.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:53:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:53:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:53:14.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:53:14.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:53:14.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:53:14.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:53:14.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:53:14.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:53:14.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:53:14.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:53:14.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:53:14.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:14.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:53:14.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:53:14.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:53:14.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:53:14.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:53:14.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:53:14.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:14.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:53:14.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.277516 2024-04-25T08:53:14.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:53:14.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:53:14.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:53:14.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:53:14.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizeup td/osd-recovery-stats 2024-04-25T08:53:14.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:186: TEST_recovery_sizeup: local dir=td/osd-recovery-stats 2024-04-25T08:53:14.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:188: TEST_recovery_sizeup: run_mon td/osd-recovery-stats a 2024-04-25T08:53:14.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:53:14.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:53:14.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:53:14.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:53:14.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:53:14.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:53:15.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:53:15.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:53:15.376 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:53:15.377 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:53:15.377 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:15.377 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:53:15.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:53:15.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:53:15.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:53:15.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:53:15.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:53:15.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:53:15.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:53:15.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:53:15.591 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:53:15.591 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:53:15.591 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:53:15.592 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:53:15.592 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:15.593 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:53:15.593 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:53:15.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:53:15.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get fsid 2024-04-25T08:53:15.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:53:15.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:53:15.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:53:15.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:53:15.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:53:15.708 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:53:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:53:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:53:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:53:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:53:15.709 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:53:15.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:53:15.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get mon_host 2024-04-25T08:53:15.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:189: TEST_recovery_sizeup: run_mgr td/osd-recovery-stats x 2024-04-25T08:53:15.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:53:15.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:53:15.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:53:15.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:53:15.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:53:15.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:53:16.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:53:16.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:53:16.714 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:53:16.715 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:53:16.715 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:16.715 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:53:16.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:53:16.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:53:16.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:53:16.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:190: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 0 2024-04-25T08:53:16.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:53:16.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:53:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:53:16.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:53:16.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:53:16.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:53:16.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:53:16.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:53:16.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:53:16.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:53:16.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:53:16.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:53:16.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a5a6ecaa-6d7d-4e75-8f63-acb3979ad1ad 2024-04-25T08:53:16.756 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 a5a6ecaa-6d7d-4e75-8f63-acb3979ad1ad 2024-04-25T08:53:16.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a5a6ecaa-6d7d-4e75-8f63-acb3979ad1ad' 2024-04-25T08:53:16.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:53:16.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8GSpmqwPDLhAAS6OKr7yDYIMajs1pCVHUjw== 2024-04-25T08:53:16.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8GSpmqwPDLhAAS6OKr7yDYIMajs1pCVHUjw=="}' 2024-04-25T08:53:16.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a5a6ecaa-6d7d-4e75-8f63-acb3979ad1ad -i td/osd-recovery-stats/0/new.json 2024-04-25T08:53:17.090 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:53:17.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:53:17.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8GSpmqwPDLhAAS6OKr7yDYIMajs1pCVHUjw== --osd-uuid a5a6ecaa-6d7d-4e75-8f63-acb3979ad1ad 2024-04-25T08:53:17.140 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:17.150+0000 7f0d9338c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:17.167 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:17.178+0000 7f0d9338c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:17.169 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:17.178+0000 7f0d9338c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:17.169 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:17.178+0000 7f0d9338c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:53:17.169 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:17.178+0000 7f0d9338c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:53:17.169 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:17.178+0000 7f0d9338c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:53:17.207 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:17.218+0000 7f0d9338c7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:53:23.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:53:23.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:53:23.659 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:53:23.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:53:23.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:53:24.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:53:24.220 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:53:24.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:53:24.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:53:24.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:53:24.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:53:24.251 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:24.262+0000 7f314b5497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:24.253 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:24.262+0000 7f314b5497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:24.254 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:24.266+0000 7f314b5497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:24.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:53:24.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:53:24.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:53:24.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:53:24.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:53:24.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:24.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:53:24.676 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:53:24.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:24.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:24.800 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:24.810+0000 7f314b5497c0 -1 Falling back to public interface 2024-04-25T08:53:25.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:26.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:26.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:26.132 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:53:26.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:53:26.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:26.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:26.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:27.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:27.450+0000 7f314b5497c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:53:27.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:27.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:27.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:53:27.531 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:53:27.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:27.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:28.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:29.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:29.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:29.193 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:53:29.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:53:29.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:29.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:29.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:30.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:30.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:30.660 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:53:30.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:53:30.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:30.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:31.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:32.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:32.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:32.106 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:53:32.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:53:32.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:32.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:32.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:33.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:33.629 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:53:33.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:53:33.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:33.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:34.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:35.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:35.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:35.058 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:53:35.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:53:35.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:35.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:35.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:36.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:36.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:36.626 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:53:36.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:53:36.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:36.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:37.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:38.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:38.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:38.061 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:53:38.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:53:38.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:38.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:38.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:39.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:39.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:39.660 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:53:39.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:53:39.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:39.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:40.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:41.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:41.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:41.093 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:53:41.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:53:41.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:41.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:42.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:43.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:43.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:43.226 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:53:43.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:53:43.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:43.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:44.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:45.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:45.422 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:53:45.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:53:45.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:45.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:46.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:47.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:47.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:47.652 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:53:47.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:53:47.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:47.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:48.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:49.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:49.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:49.575 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:53:49.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:53:49.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:49.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:50.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:51.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:51.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:51.713 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:53:51.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:53:51.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:51.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:52.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:53.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:53.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:53.454 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:53:53.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:53:53.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:53.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:53:53.922 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2631168280,v1:127.0.0.1:6803/2631168280] [v2:127.0.0.1:6804/2631168280,v1:127.0.0.1:6805/2631168280] exists,up a5a6ecaa-6d7d-4e75-8f63-acb3979ad1ad 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:191: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 1 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:53:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:53:53.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:53:53.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:53:53.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:53:53.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:53:53.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:53:53.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:53:53.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:53:53.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:53:53.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:53:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:53:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:53:53.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:53:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:53:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:53:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:53:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:53:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:53:53.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:53:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:53:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:53:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:53:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:53:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:53:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:53:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:53:53.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:53:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7a99807b-4d77-43d5-b765-80e51ac581f8 2024-04-25T08:53:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7a99807b-4d77-43d5-b765-80e51ac581f8' 2024-04-25T08:53:53.929 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 7a99807b-4d77-43d5-b765-80e51ac581f8 2024-04-25T08:53:53.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:53:53.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAhGipmYu4YORAAuFSfnH3amdjvBiO62wddoA== 2024-04-25T08:53:53.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAhGipmYu4YORAAuFSfnH3amdjvBiO62wddoA=="}' 2024-04-25T08:53:53.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7a99807b-4d77-43d5-b765-80e51ac581f8 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:53:54.442 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:53:54.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:53:54.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhGipmYu4YORAAuFSfnH3amdjvBiO62wddoA== --osd-uuid 7a99807b-4d77-43d5-b765-80e51ac581f8 2024-04-25T08:53:54.492 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:54.501+0000 7fda786327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:54.519 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:54.529+0000 7fda786327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:54.520 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:54.529+0000 7fda786327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:54.521 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:54.529+0000 7fda786327c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:53:54.521 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:54.529+0000 7fda786327c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:53:54.521 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:54.529+0000 7fda786327c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:53:54.558 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:53:54.565+0000 7fda786327c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:54:01.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:54:01.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:54:01.354 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T08:54:01.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:54:01.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:54:01.925 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:54:01.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:54:01.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:54:01.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:54:01.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:54:01.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:54:01.951 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:01.957+0000 7ff6e35117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:01.953 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:01.961+0000 7ff6e35117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:01.954 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:01.961+0000 7ff6e35117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:02.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:54:02.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:54:02.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:54:02.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:54:02.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:54:02.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:02.419 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:54:02.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:54:02.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:02.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:02.508 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:02.517+0000 7ff6e35117c0 -1 Falling back to public interface 2024-04-25T08:54:02.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:03.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:03.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:03.863 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:54:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:54:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:03.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:04.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:05.098 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:05.105+0000 7ff6e35117c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:54:05.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:05.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:05.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:54:05.279 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:54:05.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:05.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:05.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:06.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:06.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:06.947 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:54:06.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:54:06.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:06.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:07.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:08.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:08.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:08.532 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:54:08.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:54:08.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:08.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:08.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:09.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:09.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:09.940 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:54:09.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:54:09.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:09.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:10.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:11.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:11.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:11.498 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:54:11.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:54:11.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:11.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:11.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:12.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:12.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:12.921 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:54:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:54:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:12.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:13.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:14.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:14.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:14.520 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:54:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:54:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:14.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:15.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:15.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:15.948 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:54:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:54:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:15.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:16.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:17.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:17.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:17.672 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:54:17.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:54:17.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:17.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:18.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:19.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:19.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:19.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:54:19.253 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:54:19.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:19.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:19.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:20.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:20.977 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:54:20.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:54:20.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:20.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:21.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:22.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:22.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:22.554 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:54:22.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:54:22.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:22.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:23.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:24.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:24.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:24.242 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:54:24.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:54:24.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:24.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:25.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:26.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:26.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:26.136 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:54:26.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:54:26.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:26.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:27.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:28.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:28.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:28.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:54:28.173 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:54:28.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:28.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:29.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:30.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:30.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:30.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:54:30.037 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:54:30.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:30.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:30.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:31.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:31.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:31.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:54:31.960 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:54:31.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:31.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:54:32.384 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3735414694,v1:127.0.0.1:6811/3735414694] [v2:127.0.0.1:6812/3735414694,v1:127.0.0.1:6813/3735414694] exists,up 7a99807b-4d77-43d5-b765-80e51ac581f8 2024-04-25T08:54:32.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:54:32.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:54:32.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:54:32.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:192: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 2 2024-04-25T08:54:32.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:54:32.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:54:32.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:54:32.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:54:32.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:54:32.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:54:32.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:54:32.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:54:32.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:54:32.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:54:32.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1541ba23-4a55-408f-b798-7c832aaf30a6 2024-04-25T08:54:32.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 1541ba23-4a55-408f-b798-7c832aaf30a6' 2024-04-25T08:54:32.392 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 1541ba23-4a55-408f-b798-7c832aaf30a6 2024-04-25T08:54:32.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:54:32.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIGipmktVWGRAAwaNRLxxl/b9k2Fyvg/m66w== 2024-04-25T08:54:32.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIGipmktVWGRAAwaNRLxxl/b9k2Fyvg/m66w=="}' 2024-04-25T08:54:32.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1541ba23-4a55-408f-b798-7c832aaf30a6 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:54:32.904 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:54:32.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:54:32.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIGipmktVWGRAAwaNRLxxl/b9k2Fyvg/m66w== --osd-uuid 1541ba23-4a55-408f-b798-7c832aaf30a6 2024-04-25T08:54:32.959 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:32.968+0000 7fe2a223b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:32.999 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:33.008+0000 7fe2a223b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:33.000 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:33.008+0000 7fe2a223b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:33.001 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:33.008+0000 7fe2a223b7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:54:33.001 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:33.008+0000 7fe2a223b7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:54:33.001 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:33.008+0000 7fe2a223b7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:54:33.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:33.044+0000 7fe2a223b7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:54:39.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:54:39.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:54:39.786 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T08:54:39.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:54:39.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:54:40.372 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T08:54:40.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:54:40.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:54:40.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:54:40.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:54:40.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:54:40.404 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:40.412+0000 7f997a3097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:40.406 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:40.412+0000 7f997a3097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:40.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:40.416+0000 7f997a3097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:40.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:54:40.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:54:40.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:54:40.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:54:40.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:54:40.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:40.774 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:54:40.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:54:40.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:40.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:40.976 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:40.984+0000 7f997a3097c0 -1 Falling back to public interface 2024-04-25T08:54:41.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:42.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:42.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:42.181 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:54:42.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:54:42.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:42.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:42.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:43.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:43.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:43.599 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:54:43.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:54:43.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:43.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:43.670 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:54:43.676+0000 7f997a3097c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:54:44.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:45.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:45.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:45.031 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:54:45.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:54:45.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:45.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:45.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:46.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:46.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:46.661 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:54:46.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:54:46.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:46.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:47.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:48.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:48.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:48.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:54:48.099 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:54:48.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:48.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:48.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:49.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:49.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:49.694 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:54:49.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:54:49.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:49.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:50.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:51.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:51.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:51.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:54:51.250 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:54:51.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:51.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:51.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:52.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:52.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:52.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:54:52.839 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:54:52.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:52.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:53.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:54.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:54.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:54.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:54:54.270 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:54:54.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:54.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:54.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:55.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:55.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:55.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:54:55.806 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:54:55.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:55.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:56.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:57.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:57.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:57.307 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:54:57.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:54:57.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:57.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:57.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:58.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:58.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:58.876 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:54:58.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:54:58.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:58.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:54:59.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:00.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:00.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:00.356 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:55:00.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:55:00.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:00.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:55:01.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:02.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:02.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:02.255 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:55:02.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:55:02.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:02.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:55:03.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:04.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:04.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:04.170 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:55:04.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:55:04.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:04.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:55:05.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:06.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:06.222 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:55:06.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:55:06.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:06.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:55:06.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:07.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:07.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:07.799 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:55:07.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:55:07.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:07.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:55:08.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:09.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:09.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:09.882 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:55:09.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:55:09.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:09.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:55:10.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:11.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:11.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:11.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:55:11.837 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:55:11.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:11.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:55:12.286 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1706863186,v1:127.0.0.1:6819/1706863186] [v2:127.0.0.1:6820/1706863186,v1:127.0.0.1:6821/1706863186] exists,up 1541ba23-4a55-408f-b798-7c832aaf30a6 2024-04-25T08:55:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:55:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:55:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:55:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:193: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 3 2024-04-25T08:55:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:55:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:55:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:55:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:55:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:55:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:55:12.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:55:12.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:55:12.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:55:12.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bf8e4abd-e763-45f5-9c45-86ed7c372093 2024-04-25T08:55:12.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 bf8e4abd-e763-45f5-9c45-86ed7c372093' 2024-04-25T08:55:12.294 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 bf8e4abd-e763-45f5-9c45-86ed7c372093 2024-04-25T08:55:12.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:55:12.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwGipmPXZmExAAug1rjtZMhWzQcQkZnk52LQ== 2024-04-25T08:55:12.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwGipmPXZmExAAug1rjtZMhWzQcQkZnk52LQ=="}' 2024-04-25T08:55:12.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bf8e4abd-e763-45f5-9c45-86ed7c372093 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:55:12.769 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:55:12.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:55:12.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwGipmPXZmExAAug1rjtZMhWzQcQkZnk52LQ== --osd-uuid bf8e4abd-e763-45f5-9c45-86ed7c372093 2024-04-25T08:55:12.819 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:12.831+0000 7fbb48de67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:12.838 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:12.847+0000 7fbb48de67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:12.839 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:12.851+0000 7fbb48de67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:12.840 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:12.851+0000 7fbb48de67c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:55:12.840 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:12.851+0000 7fbb48de67c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:55:12.840 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:12.851+0000 7fbb48de67c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:55:12.894 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:12.903+0000 7fbb48de67c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:55:19.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:55:19.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:55:19.609 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T08:55:19.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:55:19.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:55:20.179 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T08:55:20.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:55:20.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:55:20.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:55:20.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:55:20.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:55:20.211 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:20.223+0000 7f859a0637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:20.213 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:20.223+0000 7f859a0637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:20.214 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:20.223+0000 7f859a0637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:20.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:55:20.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:55:20.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:55:20.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:55:20.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:55:20.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:20.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:55:20.585 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:55:20.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:20.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:20.768 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:20.779+0000 7f859a0637c0 -1 Falling back to public interface 2024-04-25T08:55:21.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:22.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:22.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:22.016 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:55:22.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:55:22.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:22.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:22.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:23.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:23.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:23.431 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:55:23.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:55:23.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:23.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:23.451+0000 7f859a0637c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:55:23.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:24.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:24.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:24.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:55:24.855 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:55:24.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:24.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:25.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:26.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:26.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:26.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:55:26.370 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:55:26.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:26.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:26.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:27.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:27.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:27.932 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:55:27.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:55:27.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:27.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:28.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:29.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:29.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:29.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:55:29.536 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:55:29.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:29.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:30.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:31.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:31.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:31.135 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:55:31.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:55:31.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:31.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:31.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:32.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:32.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:32.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:55:32.772 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:55:32.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:32.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:33.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:34.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:34.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:34.365 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:55:34.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:55:34.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:34.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:34.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:35.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:35.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:35.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:55:35.858 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:55:35.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:35.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:36.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:37.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:37.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:37.289 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:55:37.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:55:37.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:37.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:37.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:38.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:38.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:38.862 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:55:38.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:55:38.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:38.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:39.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:40.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:40.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:40.521 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:55:40.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:55:40.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:40.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:41.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:42.397 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:55:42.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:55:42.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:42.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:43.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:44.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:44.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:44.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:55:44.148 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:55:44.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:44.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:45.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:46.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:46.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:46.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:55:46.140 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:55:46.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:46.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:46.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:47.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:47.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:47.724 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:55:47.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:55:47.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:47.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:48.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:49.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:49.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:49.858 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:55:49.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:55:49.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:49.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:55:50.267 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1159411287,v1:127.0.0.1:6827/1159411287] [v2:127.0.0.1:6828/1159411287,v1:127.0.0.1:6829/1159411287] exists,up bf8e4abd-e763-45f5-9c45-86ed7c372093 2024-04-25T08:55:50.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:55:50.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:55:50.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:55:50.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:194: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 4 2024-04-25T08:55:50.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:55:50.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:55:50.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:55:50.271 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:55:50.272 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:55:50.272 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:55:50.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:55:50.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:55:50.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:55:50.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:55:50.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:55:50.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:55:50.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:55:50.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:55:50.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:55:50.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:55:50.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:55:50.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:55:50.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:55:50.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:55:50.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:55:50.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=74344132-b4e9-4531-ae63-a137df18c7f6 2024-04-25T08:55:50.275 INFO:tasks.workunit.client.0.smithi102.stdout:add osd4 74344132-b4e9-4531-ae63-a137df18c7f6 2024-04-25T08:55:50.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 74344132-b4e9-4531-ae63-a137df18c7f6' 2024-04-25T08:55:50.275 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:55:50.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCWGipmDZ5TEhAAIndDHDI7x4ee7hCZufMRiA== 2024-04-25T08:55:50.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCWGipmDZ5TEhAAIndDHDI7x4ee7hCZufMRiA=="}' 2024-04-25T08:55:50.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 74344132-b4e9-4531-ae63-a137df18c7f6 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:55:50.883 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:55:50.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:55:50.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWGipmDZ5TEhAAIndDHDI7x4ee7hCZufMRiA== --osd-uuid 74344132-b4e9-4531-ae63-a137df18c7f6 2024-04-25T08:55:50.934 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:50.942+0000 7f7de734d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:50.963 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:50.974+0000 7f7de734d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:50.964 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:50.974+0000 7f7de734d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:50.965 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:50.974+0000 7f7de734d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:55:50.965 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:50.974+0000 7f7de734d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:55:50.965 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:50.974+0000 7f7de734d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:55:51.002 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:51.010+0000 7f7de734d7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:55:58.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:55:58.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:55:58.008 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd4 key to auth repository 2024-04-25T08:55:58.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:55:58.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:55:58.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:55:58.547 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.4 2024-04-25T08:55:58.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:55:58.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:55:58.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:55:58.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:55:58.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:58.590+0000 7f2f2e7677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:58.581 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:58.590+0000 7f2f2e7677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:58.582 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:58.594+0000 7f2f2e7677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:55:58.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:55:58.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:55:58.935 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:55:58.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:55:58.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:55:58.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:55:58.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:58.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:55:58.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:58.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:55:59.140 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:55:59.150+0000 7f2f2e7677c0 -1 Falling back to public interface 2024-04-25T08:55:59.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:00.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:00.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:00.381 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:56:00.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:56:00.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:00.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:00.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:01.768 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:01.778+0000 7f2f2e7677c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:56:01.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:01.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:01.800 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:56:01.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:56:01.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:01.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:02.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:03.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:03.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:03.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:56:03.368 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:56:03.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:03.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:03.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:04.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:04.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:04.929 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:56:04.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:56:04.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:04.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:05.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:06.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:06.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:06.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:56:06.387 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:56:06.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:06.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:06.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:07.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:07.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:07.918 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:56:07.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:56:07.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:07.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:08.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:09.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:09.471 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:56:09.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:56:09.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:09.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:09.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:10.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:10.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:10.895 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:56:10.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:56:10.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:10.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:11.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:12.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:12.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:12.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:56:12.347 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:56:12.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:12.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:12.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:13.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:13.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:13.936 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:56:13.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:56:13.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:13.937 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:14.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:15.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:15.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:15.445 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:56:15.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:56:15.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:15.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:15.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:16.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:16.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:16.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:56:16.911 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:56:16.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:16.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:17.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:18.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:18.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:18.336 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:56:18.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:56:18.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:18.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:19.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:20.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:20.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:20.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:56:20.525 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:56:20.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:20.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:21.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:22.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:22.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:22.503 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:56:22.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:56:22.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:22.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:23.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:24.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:24.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:24.304 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:56:24.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:56:24.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:24.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:24.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:25.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:25.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:25.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:56:25.979 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:56:25.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:25.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:26.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:27.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:27.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:27.764 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:56:27.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:56:27.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:27.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:28.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:29.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:29.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:29.565 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:56:29.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:56:29.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:29.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:56:29.991 INFO:tasks.workunit.client.0.smithi102.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/325999391,v1:127.0.0.1:6835/325999391] [v2:127.0.0.1:6836/325999391,v1:127.0.0.1:6837/325999391] exists,up 74344132-b4e9-4531-ae63-a137df18c7f6 2024-04-25T08:56:29.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:195: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 5 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:56:29.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:56:29.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:56:29.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:56:29.998 INFO:tasks.workunit.client.0.smithi102.stdout:add osd5 8a272b88-511e-4549-a2bf-041ef6094d56 2024-04-25T08:56:29.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8a272b88-511e-4549-a2bf-041ef6094d56 2024-04-25T08:56:29.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 8a272b88-511e-4549-a2bf-041ef6094d56' 2024-04-25T08:56:29.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:56:30.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC+GipmY+e4ARAAFgPu50V2ii4X9bh9wGsmOw== 2024-04-25T08:56:30.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC+GipmY+e4ARAAFgPu50V2ii4X9bh9wGsmOw=="}' 2024-04-25T08:56:30.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8a272b88-511e-4549-a2bf-041ef6094d56 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:56:30.525 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:56:30.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:56:30.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC+GipmY+e4ARAAFgPu50V2ii4X9bh9wGsmOw== --osd-uuid 8a272b88-511e-4549-a2bf-041ef6094d56 2024-04-25T08:56:30.570 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:30.577+0000 7f8c51f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:30.588 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:30.597+0000 7f8c51f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:30.589 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:30.597+0000 7f8c51f0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:30.590 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:30.597+0000 7f8c51f0d7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:56:30.590 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:30.597+0000 7f8c51f0d7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:56:30.590 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:30.597+0000 7f8c51f0d7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:56:30.641 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:30.649+0000 7f8c51f0d7c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:56:37.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:56:37.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:56:37.577 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd5 key to auth repository 2024-04-25T08:56:37.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:56:37.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:56:38.149 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.5 2024-04-25T08:56:38.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:56:38.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:56:38.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:56:38.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:56:38.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:56:38.180 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:38.189+0000 7f0cffdef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:38.182 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:38.193+0000 7f0cffdef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:38.183 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:38.193+0000 7f0cffdef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:38.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:56:38.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:56:38.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:56:38.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:56:38.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:56:38.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:38.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:56:38.607 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:56:38.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:38.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:38.756 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:38.765+0000 7f0cffdef7c0 -1 Falling back to public interface 2024-04-25T08:56:39.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:40.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:40.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:40.057 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:56:40.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:56:40.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:40.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:40.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:41.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:41.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:41.476 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:56:41.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:56:41.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:41.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:41.480 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:56:41.489+0000 7f0cffdef7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:56:41.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:42.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:42.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:42.892 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:56:42.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:56:42.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:42.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:43.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:44.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:44.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:44.311 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:56:44.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:56:44.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:44.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:44.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:45.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:45.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:45.772 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:56:45.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:56:45.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:45.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:46.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:47.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:47.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:47.328 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:56:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:56:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:47.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:47.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:48.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:48.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:48.869 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:56:48.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:56:48.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:48.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:49.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:50.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:50.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:50.568 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:56:50.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:56:50.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:50.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:51.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:52.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:52.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:52.167 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:56:52.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:56:52.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:52.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:52.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:53.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:53.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:53.591 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:56:53.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:56:53.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:53.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:54.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:55.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:55.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:55.151 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:56:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:56:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:55.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:55.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:56.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:56.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:56.657 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:56:56.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:56:56.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:56.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:57.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:58.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:58.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:58.216 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:56:58.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:56:58.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:58.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:56:59.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:00.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:00.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:00.385 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:57:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:57:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:00.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:57:01.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:02.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:02.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:02.212 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:57:02.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:57:02.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:02.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:57:02.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:03.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:03.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:03.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:57:03.955 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:57:03.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:03.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:57:04.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:05.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:05.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:05.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:57:05.819 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:57:05.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:05.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:57:06.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:07.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:07.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:07.448 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T08:57:07.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:57:07.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:07.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:57:08.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:09.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:09.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:09.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:57:09.070 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T08:57:09.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:09.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:57:09.521 INFO:tasks.workunit.client.0.smithi102.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2276578761,v1:127.0.0.1:6843/2276578761] [v2:127.0.0.1:6844/2276578761,v1:127.0.0.1:6845/2276578761] exists,up 8a272b88-511e-4549-a2bf-041ef6094d56 2024-04-25T08:57:09.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:57:09.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:57:09.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:57:09.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:197: TEST_recovery_sizeup: create_pool test 1 1 2024-04-25T08:57:09.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:57:10.066 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T08:57:10.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:57:11.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:198: TEST_recovery_sizeup: ceph osd pool set test size 2 2024-04-25T08:57:12.334 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-04-25T08:57:12.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:200: TEST_recovery_sizeup: wait_for_clean 2024-04-25T08:57:12.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:57:12.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:57:12.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:57:12.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:57:12.360 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:57:12.360 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:57:12.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:57:12.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:57:12.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:57:12.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:57:12.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:57:12.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:57:12.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:57:12.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:57:12.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:57:13.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:57:13.110 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:57:13.110 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:57:13.110 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:57:13.110 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T08:57:13.110 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T08:57:13.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:57:13.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:13.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:57:13.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836530 2024-04-25T08:57:13.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836530 2024-04-25T08:57:13.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530' 2024-04-25T08:57:13.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:13.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:57:13.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672997 2024-04-25T08:57:13.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672997 2024-04-25T08:57:13.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949672997' 2024-04-25T08:57:13.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:13.401 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:57:13.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509471 2024-04-25T08:57:13.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509471 2024-04-25T08:57:13.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949672997 2-64424509471' 2024-04-25T08:57:13.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:13.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:57:13.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345943 2024-04-25T08:57:13.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345943 2024-04-25T08:57:13.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949672997 2-64424509471 3-85899345943' 2024-04-25T08:57:13.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:13.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:57:13.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182419 2024-04-25T08:57:13.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182419 2024-04-25T08:57:13.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949672997 2-64424509471 3-85899345943 4-107374182419' 2024-04-25T08:57:13.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:13.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:57:14.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018888 2024-04-25T08:57:14.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018888 2024-04-25T08:57:14.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949672997 2-64424509471 3-85899345943 4-107374182419 5-128849018888' 2024-04-25T08:57:14.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:14.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:14.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836530 2024-04-25T08:57:14.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:57:14.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836530 2024-04-25T08:57:14.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:14.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836530 2024-04-25T08:57:14.021 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836530 2024-04-25T08:57:14.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-04-25T08:57:14.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:57:14.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836530 -lt 21474836530 2024-04-25T08:57:14.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:14.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672997 2024-04-25T08:57:14.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:14.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:57:14.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672997 2024-04-25T08:57:14.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:14.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672997 2024-04-25T08:57:14.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2024-04-25T08:57:14.439 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672997 2024-04-25T08:57:14.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:57:14.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672995 -lt 42949672997 2024-04-25T08:57:14.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:57:15.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:57:15.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:57:16.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672997 -lt 42949672997 2024-04-25T08:57:16.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:16.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:16.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509471 2024-04-25T08:57:16.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:57:16.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509471 2024-04-25T08:57:16.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:16.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509471 2024-04-25T08:57:16.330 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509471 2024-04-25T08:57:16.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-04-25T08:57:16.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:57:16.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509471 -lt 64424509471 2024-04-25T08:57:16.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:16.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345943 2024-04-25T08:57:16.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:16.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:57:16.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345943 2024-04-25T08:57:16.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:16.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345943 2024-04-25T08:57:16.777 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345943 2024-04-25T08:57:16.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345943' 2024-04-25T08:57:16.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:57:17.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345944 -lt 85899345943 2024-04-25T08:57:17.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:17.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:17.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182419 2024-04-25T08:57:17.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:57:17.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182419 2024-04-25T08:57:17.225 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:17.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182419 2024-04-25T08:57:17.227 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182419 2024-04-25T08:57:17.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182419' 2024-04-25T08:57:17.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:57:17.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182420 -lt 107374182419 2024-04-25T08:57:17.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:17.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:17.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018888 2024-04-25T08:57:17.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:57:17.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018888 2024-04-25T08:57:17.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:17.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018888 2024-04-25T08:57:17.691 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018888 2024-04-25T08:57:17.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2024-04-25T08:57:17.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:57:18.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018889 -lt 128849018888 2024-04-25T08:57:18.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:57:18.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:57:18.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:57:18.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:57:18.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:57:18.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:57:18.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:57:18.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:57:18.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:57:18.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:57:18.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:57:19.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:57:19.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:57:19.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:57:19.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:57:19.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:57:19.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:57:19.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:57:19.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: seq 1 200 2024-04-25T08:57:19.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:19.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj1 /dev/null 2024-04-25T08:57:19.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:19.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj2 /dev/null 2024-04-25T08:57:19.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:19.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj3 /dev/null 2024-04-25T08:57:20.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:20.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj4 /dev/null 2024-04-25T08:57:20.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:20.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj5 /dev/null 2024-04-25T08:57:20.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:20.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj6 /dev/null 2024-04-25T08:57:20.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:20.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj7 /dev/null 2024-04-25T08:57:20.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:20.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj8 /dev/null 2024-04-25T08:57:20.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:20.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj9 /dev/null 2024-04-25T08:57:20.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:20.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj10 /dev/null 2024-04-25T08:57:20.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:20.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj11 /dev/null 2024-04-25T08:57:21.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:21.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj12 /dev/null 2024-04-25T08:57:21.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:21.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj13 /dev/null 2024-04-25T08:57:21.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:21.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj14 /dev/null 2024-04-25T08:57:21.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:21.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj15 /dev/null 2024-04-25T08:57:21.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:21.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj16 /dev/null 2024-04-25T08:57:21.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:21.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj17 /dev/null 2024-04-25T08:57:22.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:22.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj18 /dev/null 2024-04-25T08:57:22.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:22.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj19 /dev/null 2024-04-25T08:57:22.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:22.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj20 /dev/null 2024-04-25T08:57:22.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:22.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj21 /dev/null 2024-04-25T08:57:22.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:22.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj22 /dev/null 2024-04-25T08:57:22.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:22.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj23 /dev/null 2024-04-25T08:57:22.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:22.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj24 /dev/null 2024-04-25T08:57:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj25 /dev/null 2024-04-25T08:57:23.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:23.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj26 /dev/null 2024-04-25T08:57:23.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:23.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj27 /dev/null 2024-04-25T08:57:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:23.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj28 /dev/null 2024-04-25T08:57:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:23.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj29 /dev/null 2024-04-25T08:57:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:23.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj30 /dev/null 2024-04-25T08:57:23.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:23.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj31 /dev/null 2024-04-25T08:57:23.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:23.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj32 /dev/null 2024-04-25T08:57:23.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:23.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj33 /dev/null 2024-04-25T08:57:24.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:24.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj34 /dev/null 2024-04-25T08:57:24.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:24.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj35 /dev/null 2024-04-25T08:57:24.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:24.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj36 /dev/null 2024-04-25T08:57:24.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:24.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj37 /dev/null 2024-04-25T08:57:24.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:24.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj38 /dev/null 2024-04-25T08:57:24.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:24.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj39 /dev/null 2024-04-25T08:57:24.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:24.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj40 /dev/null 2024-04-25T08:57:24.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:24.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj41 /dev/null 2024-04-25T08:57:25.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:25.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj42 /dev/null 2024-04-25T08:57:25.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:25.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj43 /dev/null 2024-04-25T08:57:25.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:25.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj44 /dev/null 2024-04-25T08:57:25.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:25.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj45 /dev/null 2024-04-25T08:57:25.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:25.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj46 /dev/null 2024-04-25T08:57:25.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:25.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj47 /dev/null 2024-04-25T08:57:25.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:25.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj48 /dev/null 2024-04-25T08:57:25.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:25.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj49 /dev/null 2024-04-25T08:57:26.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:26.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj50 /dev/null 2024-04-25T08:57:26.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:26.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj51 /dev/null 2024-04-25T08:57:26.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:26.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj52 /dev/null 2024-04-25T08:57:26.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:26.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj53 /dev/null 2024-04-25T08:57:26.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:26.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj54 /dev/null 2024-04-25T08:57:26.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:26.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj55 /dev/null 2024-04-25T08:57:26.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:26.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj56 /dev/null 2024-04-25T08:57:26.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:26.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj57 /dev/null 2024-04-25T08:57:27.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:27.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj58 /dev/null 2024-04-25T08:57:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:27.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj59 /dev/null 2024-04-25T08:57:27.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:27.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj60 /dev/null 2024-04-25T08:57:27.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:27.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj61 /dev/null 2024-04-25T08:57:27.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:27.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj62 /dev/null 2024-04-25T08:57:27.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:27.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj63 /dev/null 2024-04-25T08:57:27.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:27.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj64 /dev/null 2024-04-25T08:57:28.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:28.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj65 /dev/null 2024-04-25T08:57:28.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:28.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj66 /dev/null 2024-04-25T08:57:28.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:28.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj67 /dev/null 2024-04-25T08:57:28.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:28.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj68 /dev/null 2024-04-25T08:57:28.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:28.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj69 /dev/null 2024-04-25T08:57:28.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:28.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj70 /dev/null 2024-04-25T08:57:28.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:28.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj71 /dev/null 2024-04-25T08:57:28.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:28.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj72 /dev/null 2024-04-25T08:57:28.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:28.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj73 /dev/null 2024-04-25T08:57:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:29.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj74 /dev/null 2024-04-25T08:57:29.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:29.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj75 /dev/null 2024-04-25T08:57:29.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:29.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj76 /dev/null 2024-04-25T08:57:29.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:29.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj77 /dev/null 2024-04-25T08:57:29.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:29.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj78 /dev/null 2024-04-25T08:57:29.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:29.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj79 /dev/null 2024-04-25T08:57:29.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:29.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj80 /dev/null 2024-04-25T08:57:29.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:29.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj81 /dev/null 2024-04-25T08:57:30.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:30.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj82 /dev/null 2024-04-25T08:57:30.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:30.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj83 /dev/null 2024-04-25T08:57:30.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:30.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj84 /dev/null 2024-04-25T08:57:30.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:30.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj85 /dev/null 2024-04-25T08:57:30.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:30.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj86 /dev/null 2024-04-25T08:57:30.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:30.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj87 /dev/null 2024-04-25T08:57:30.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:30.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj88 /dev/null 2024-04-25T08:57:30.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:30.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj89 /dev/null 2024-04-25T08:57:31.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:31.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj90 /dev/null 2024-04-25T08:57:31.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:31.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj91 /dev/null 2024-04-25T08:57:31.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:31.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj92 /dev/null 2024-04-25T08:57:31.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:31.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj93 /dev/null 2024-04-25T08:57:31.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:31.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj94 /dev/null 2024-04-25T08:57:31.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:31.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj95 /dev/null 2024-04-25T08:57:31.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:31.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj96 /dev/null 2024-04-25T08:57:31.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:31.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj97 /dev/null 2024-04-25T08:57:32.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:32.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj98 /dev/null 2024-04-25T08:57:32.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:32.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj99 /dev/null 2024-04-25T08:57:32.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:32.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj100 /dev/null 2024-04-25T08:57:32.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:32.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj101 /dev/null 2024-04-25T08:57:32.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:32.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj102 /dev/null 2024-04-25T08:57:32.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:32.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj103 /dev/null 2024-04-25T08:57:32.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:32.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj104 /dev/null 2024-04-25T08:57:32.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:32.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj105 /dev/null 2024-04-25T08:57:33.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:33.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj106 /dev/null 2024-04-25T08:57:33.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:33.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj107 /dev/null 2024-04-25T08:57:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj108 /dev/null 2024-04-25T08:57:33.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:33.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj109 /dev/null 2024-04-25T08:57:33.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:33.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj110 /dev/null 2024-04-25T08:57:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:33.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj111 /dev/null 2024-04-25T08:57:33.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:33.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj112 /dev/null 2024-04-25T08:57:33.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:33.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj113 /dev/null 2024-04-25T08:57:34.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:34.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj114 /dev/null 2024-04-25T08:57:34.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:34.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj115 /dev/null 2024-04-25T08:57:34.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:34.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj116 /dev/null 2024-04-25T08:57:34.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:34.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj117 /dev/null 2024-04-25T08:57:34.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:34.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj118 /dev/null 2024-04-25T08:57:34.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:34.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj119 /dev/null 2024-04-25T08:57:34.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:34.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj120 /dev/null 2024-04-25T08:57:34.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:34.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj121 /dev/null 2024-04-25T08:57:35.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:35.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj122 /dev/null 2024-04-25T08:57:35.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:35.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj123 /dev/null 2024-04-25T08:57:35.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:35.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj124 /dev/null 2024-04-25T08:57:35.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:35.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj125 /dev/null 2024-04-25T08:57:35.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:35.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj126 /dev/null 2024-04-25T08:57:35.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:35.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj127 /dev/null 2024-04-25T08:57:35.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:35.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj128 /dev/null 2024-04-25T08:57:35.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:35.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj129 /dev/null 2024-04-25T08:57:36.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:36.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj130 /dev/null 2024-04-25T08:57:36.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:36.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj131 /dev/null 2024-04-25T08:57:36.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:36.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj132 /dev/null 2024-04-25T08:57:36.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:36.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj133 /dev/null 2024-04-25T08:57:36.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:36.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj134 /dev/null 2024-04-25T08:57:36.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:36.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj135 /dev/null 2024-04-25T08:57:36.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:36.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj136 /dev/null 2024-04-25T08:57:36.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:36.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj137 /dev/null 2024-04-25T08:57:37.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:37.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj138 /dev/null 2024-04-25T08:57:37.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:37.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj139 /dev/null 2024-04-25T08:57:37.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:37.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj140 /dev/null 2024-04-25T08:57:37.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:37.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj141 /dev/null 2024-04-25T08:57:37.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:37.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj142 /dev/null 2024-04-25T08:57:37.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:37.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj143 /dev/null 2024-04-25T08:57:37.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:37.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj144 /dev/null 2024-04-25T08:57:37.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:37.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj145 /dev/null 2024-04-25T08:57:38.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:38.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj146 /dev/null 2024-04-25T08:57:38.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:38.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj147 /dev/null 2024-04-25T08:57:38.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:38.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj148 /dev/null 2024-04-25T08:57:38.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:38.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj149 /dev/null 2024-04-25T08:57:38.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:38.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj150 /dev/null 2024-04-25T08:57:38.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:38.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj151 /dev/null 2024-04-25T08:57:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:38.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj152 /dev/null 2024-04-25T08:57:38.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:38.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj153 /dev/null 2024-04-25T08:57:38.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:38.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj154 /dev/null 2024-04-25T08:57:39.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:39.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj155 /dev/null 2024-04-25T08:57:39.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:39.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj156 /dev/null 2024-04-25T08:57:39.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:39.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj157 /dev/null 2024-04-25T08:57:39.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:39.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj158 /dev/null 2024-04-25T08:57:39.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:39.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj159 /dev/null 2024-04-25T08:57:39.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:39.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj160 /dev/null 2024-04-25T08:57:39.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:39.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj161 /dev/null 2024-04-25T08:57:39.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:39.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj162 /dev/null 2024-04-25T08:57:40.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:40.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj163 /dev/null 2024-04-25T08:57:40.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:40.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj164 /dev/null 2024-04-25T08:57:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:40.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj165 /dev/null 2024-04-25T08:57:40.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:40.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj166 /dev/null 2024-04-25T08:57:40.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:40.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj167 /dev/null 2024-04-25T08:57:40.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:40.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj168 /dev/null 2024-04-25T08:57:40.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:40.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj169 /dev/null 2024-04-25T08:57:40.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:40.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj170 /dev/null 2024-04-25T08:57:41.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:41.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj171 /dev/null 2024-04-25T08:57:41.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:41.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj172 /dev/null 2024-04-25T08:57:41.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:41.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj173 /dev/null 2024-04-25T08:57:41.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:41.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj174 /dev/null 2024-04-25T08:57:41.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:41.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj175 /dev/null 2024-04-25T08:57:41.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:41.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj176 /dev/null 2024-04-25T08:57:41.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:41.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj177 /dev/null 2024-04-25T08:57:41.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:41.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj178 /dev/null 2024-04-25T08:57:42.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:42.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj179 /dev/null 2024-04-25T08:57:42.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:42.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj180 /dev/null 2024-04-25T08:57:42.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:42.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj181 /dev/null 2024-04-25T08:57:42.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:42.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj182 /dev/null 2024-04-25T08:57:42.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:42.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj183 /dev/null 2024-04-25T08:57:42.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:42.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj184 /dev/null 2024-04-25T08:57:42.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:42.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj185 /dev/null 2024-04-25T08:57:42.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:42.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj186 /dev/null 2024-04-25T08:57:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:43.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj187 /dev/null 2024-04-25T08:57:43.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:43.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj188 /dev/null 2024-04-25T08:57:43.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:43.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj189 /dev/null 2024-04-25T08:57:43.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:43.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj190 /dev/null 2024-04-25T08:57:43.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:43.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj191 /dev/null 2024-04-25T08:57:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:43.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj192 /dev/null 2024-04-25T08:57:43.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:43.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj193 /dev/null 2024-04-25T08:57:43.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:43.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj194 /dev/null 2024-04-25T08:57:44.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:44.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj195 /dev/null 2024-04-25T08:57:44.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:44.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj196 /dev/null 2024-04-25T08:57:44.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:44.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj197 /dev/null 2024-04-25T08:57:44.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj198 /dev/null 2024-04-25T08:57:44.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:44.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj199 /dev/null 2024-04-25T08:57:44.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T08:57:44.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj200 /dev/null 2024-04-25T08:57:44.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: get_primary test obj1 2024-04-25T08:57:44.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:57:44.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:57:44.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:57:44.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:57:45.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: local primary=1 2024-04-25T08:57:45.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: get_pg test obj1 2024-04-25T08:57:45.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:57:45.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:57:45.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:57:45.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:57:45.711 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: local PG=1.0 2024-04-25T08:57:45.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: get_not_primary test obj1 2024-04-25T08:57:45.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:57:45.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:57:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:57:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:57:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:57:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:57:45.712 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:57:46.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:57:46.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:57:46.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:57:46.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: local otherosd=0 2024-04-25T08:57:46.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:212: TEST_recovery_sizeup: ceph osd set norecover 2024-04-25T08:57:47.363 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is set 2024-04-25T08:57:47.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:213: TEST_recovery_sizeup: ceph osd out osd.1 osd.0 2024-04-25T08:57:48.168 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already out. osd.0 is already out. 2024-04-25T08:57:48.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:214: TEST_recovery_sizeup: ceph osd pool set test size 4 2024-04-25T08:57:48.828 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 4 2024-04-25T08:57:48.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:215: TEST_recovery_sizeup: ceph osd unset norecover 2024-04-25T08:57:50.229 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is unset 2024-04-25T08:57:50.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: get_primary test obj1 2024-04-25T08:57:50.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:57:50.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:57:50.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:57:50.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:57:50.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: primary=2 2024-04-25T08:57:50.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:219: TEST_recovery_sizeup: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T08:57:50.967 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:57:50.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:220: TEST_recovery_sizeup: sleep 2 2024-04-25T08:57:52.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:222: TEST_recovery_sizeup: wait_for_clean 2024-04-25T08:57:52.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:57:52.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:57:52.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:57:52.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:57:52.985 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:57:52.985 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:57:52.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:57:52.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:57:52.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:57:53.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:57:53.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:57:53.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:57:53.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:57:53.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:57:53.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:57:53.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:57:53.541 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T08:57:53.541 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T08:57:53.541 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T08:57:53.541 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T08:57:53.541 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T08:57:53.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:57:53.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:53.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:57:53.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836540 2024-04-25T08:57:53.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836540 2024-04-25T08:57:53.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540' 2024-04-25T08:57:53.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:53.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:57:53.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673007 2024-04-25T08:57:53.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673007 2024-04-25T08:57:53.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-42949673007' 2024-04-25T08:57:53.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:53.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:57:54.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509481 2024-04-25T08:57:54.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509481 2024-04-25T08:57:54.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-42949673007 2-64424509481' 2024-04-25T08:57:54.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:54.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:57:54.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345953 2024-04-25T08:57:54.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345953 2024-04-25T08:57:54.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-42949673007 2-64424509481 3-85899345953' 2024-04-25T08:57:54.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:54.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:57:54.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182430 2024-04-25T08:57:54.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182430 2024-04-25T08:57:54.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-42949673007 2-64424509481 3-85899345953 4-107374182430' 2024-04-25T08:57:54.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:57:54.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:57:54.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018899 2024-04-25T08:57:54.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018899 2024-04-25T08:57:54.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-42949673007 2-64424509481 3-85899345953 4-107374182430 5-128849018899' 2024-04-25T08:57:54.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:54.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836540 2024-04-25T08:57:54.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:54.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:57:54.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836540 2024-04-25T08:57:54.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:54.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836540 2024-04-25T08:57:54.565 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836540 2024-04-25T08:57:54.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-04-25T08:57:54.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:57:54.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836538 -lt 21474836540 2024-04-25T08:57:54.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:57:55.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:57:55.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:57:56.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836541 -lt 21474836540 2024-04-25T08:57:56.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:56.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673007 2024-04-25T08:57:56.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:56.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:57:56.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673007 2024-04-25T08:57:56.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:56.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673007 2024-04-25T08:57:56.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673007' 2024-04-25T08:57:56.441 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673007 2024-04-25T08:57:56.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:57:57.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673007 -lt 42949673007 2024-04-25T08:57:57.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:57.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:57.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509481 2024-04-25T08:57:57.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:57:57.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509481 2024-04-25T08:57:57.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:57.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509481 2024-04-25T08:57:57.088 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509481 2024-04-25T08:57:57.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-04-25T08:57:57.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:57:57.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509482 -lt 64424509481 2024-04-25T08:57:57.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:57.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345953 2024-04-25T08:57:57.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:57.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:57:57.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345953 2024-04-25T08:57:57.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:57.734 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345953 2024-04-25T08:57:57.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345953 2024-04-25T08:57:57.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345953' 2024-04-25T08:57:57.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:57:58.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345954 -lt 85899345953 2024-04-25T08:57:58.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:58.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:58.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182430 2024-04-25T08:57:58.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:57:58.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182430 2024-04-25T08:57:58.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:58.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182430 2024-04-25T08:57:58.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182430' 2024-04-25T08:57:58.256 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182430 2024-04-25T08:57:58.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:57:58.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182430 -lt 107374182430 2024-04-25T08:57:58.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:57:58.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:57:58.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018899 2024-04-25T08:57:58.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:57:58.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018899 2024-04-25T08:57:58.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:57:58.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018899 2024-04-25T08:57:58.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018899' 2024-04-25T08:57:58.750 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018899 2024-04-25T08:57:58.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:57:59.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018899 -lt 128849018899 2024-04-25T08:57:59.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:57:59.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:57:59.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:57:59.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:57:59.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:57:59.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:57:59.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:57:59.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:57:59.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:57:59.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:57:59.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:00.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:00.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:00.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:00.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:00.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:00.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:58:00.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:58:00.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:58:00.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:00.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:00.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:01.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:01.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:01.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:01.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:01.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:01.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:01.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:01.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:01.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:01.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:01.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:01.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:01.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:01.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:01.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:01.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:01.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:01.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:01.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:01.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:02.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:58:02.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:58:02.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:02.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:02.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:02.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:02.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:02.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:02.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:02.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:02.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:02.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:02.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:03.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:03.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:03.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:03.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:03.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:03.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:03.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:03.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:03.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:03.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:03.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:03.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:03.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:58:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:58:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:04.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:04.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:04.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:04.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:04.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:04.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:04.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:04.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:05.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:05.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:05.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:05.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:05.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:05.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:05.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:05.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:05.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:05.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:05.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:05.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:05.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:06.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:58:06.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:58:06.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:06.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:06.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:06.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:06.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:06.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:06.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:06.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:06.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:06.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:06.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:06.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:06.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:06.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:06.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:07.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:07.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:07.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:07.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:07.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:07.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:07.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:07.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:07.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:08.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:58:08.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:58:08.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:08.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:08.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:08.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:08.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:08.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:08.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:08.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:08.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:08.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:08.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:08.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:08.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:08.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:08.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:09.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:09.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:09.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:09.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:09.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:09.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:09.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:09.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:09.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:10.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:58:10.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:58:10.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:10.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:10.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:10.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:10.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:10.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:10.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:10.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:10.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:10.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:10.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:10.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:10.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:11.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:11.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:11.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:11.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:11.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:11.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:11.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:11.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:11.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:11.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:58:11.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:58:11.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:11.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:11.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:11.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:11.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:11.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:11.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:11.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:11.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:11.922 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:11.923 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:12.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:12.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:12.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:12.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:12.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:12.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:12.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:12.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:12.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:12.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:12.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:12.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:12.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:13.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:58:13.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:58:13.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:13.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:13.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:13.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:13.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:13.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:13.726 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:13.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:13.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:14.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:14.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:14.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:14.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:14.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:14.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:14.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:14.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:15.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:58:15.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:58:15.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:15.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:15.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:15.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:15.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:15.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:15.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:15.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:15.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:15.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:15.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:16.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:16.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:16.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:16.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:16.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:16.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:16.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:16.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:16.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:16.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:16.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:16.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:16.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:17.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:58:17.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:58:17.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:17.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:17.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:17.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:17.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:17.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:17.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:17.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:17.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:17.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:18.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:18.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:18.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:18.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:18.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:18.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:18.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:18.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:18.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:18.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:18.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:18.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:19.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:58:19.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:58:19.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:19.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:19.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:19.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:19.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:19.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:19.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:19.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:19.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:19.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:19.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:20.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:20.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:20.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:20.137 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:20.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:20.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:20.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:20.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:20.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:20.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:20.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:20.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:20.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:21.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:58:21.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:58:21.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:21.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:21.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:21.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:21.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:21.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:21.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:21.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:21.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:21.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:21.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:22.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:22.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:22.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:22.027 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:22.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:22.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:22.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:22.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:22.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:22.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:22.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:22.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:22.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:58:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:58:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:23.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:23.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:23.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:23.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:23.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:23.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:23.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:23.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:23.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:23.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:23.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:23.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:24.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:24.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:24.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:24.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:24.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:24.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:24.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:24.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:24.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:24.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:58:24.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:58:24.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:24.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:24.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:25.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:25.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:25.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:25.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:25.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:25.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:25.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:25.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:25.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:25.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:25.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:25.485 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:26.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:26.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:26.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:26.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:26.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:26.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:26.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:26.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:26.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:26.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:58:26.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:58:26.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:26.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:26.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:26.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:26.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:26.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:26.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:26.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:26.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:26.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:26.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:27.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:27.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:27.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:27.166 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:27.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:27.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:27.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:27.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:27.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:27.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:27.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:27.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:27.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:28.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:58:28.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:58:28.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:28.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:28.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:28.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:28.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:28.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:28.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:28.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:28.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:28.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:28.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:28.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:28.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:28.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:28.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:29.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:29.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:29.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:29.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:29.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:29.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:29.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:29.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:29.346 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:30.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T08:58:30.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T08:58:30.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:30.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:30.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:30.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:30.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:30.160 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:30.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:30.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:30.161 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:30.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:30.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:30.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:30.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:30.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:30.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:31.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:31.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:31.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:31.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:31.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:31.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:31.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:31.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:31.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:31.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:58:31.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:58:31.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:31.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:31.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:32.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:32.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:32.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:32.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:32.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:32.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:32.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:32.032 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:32.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:32.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:32.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:32.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:33.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:33.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:33.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:33.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:33.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:33.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:33.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:33.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:33.002 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:33.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:58:33.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:58:33.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:33.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:33.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:33.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:33.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:33.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:33.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:33.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:33.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:33.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:33.785 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:34.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:34.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:34.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:34.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:34.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:34.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:34.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:34.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:34.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:34.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:34.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:34.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:34.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:35.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T08:58:35.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T08:58:35.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:35.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:35.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:35.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:35.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:35.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:35.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:35.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:35.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:35.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:35.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:35.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:58:35.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:35.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:35.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:36.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:58:36.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:58:36.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:58:36.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:58:36.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:58:36.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:58:36.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:58:36.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:58:36.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:58:36.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2 2024-04-25T08:58:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2 '!=' null 2024-04-25T08:58:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:58:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:58:36.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:58:37.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:58:37.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:58:37.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:58:37.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:58:37.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:58:37.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:58:37.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:58:37.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:58:37.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:58:37.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:58:37.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:58:37.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:58:38.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:58:38.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:58:38.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:58:38.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: expr 200 '*' 2 2024-04-25T08:58:38.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: local degraded=400 2024-04-25T08:58:38.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: expr 200 '*' 2 2024-04-25T08:58:38.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: local misplaced=400 2024-04-25T08:58:38.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:226: TEST_recovery_sizeup: local log=td/osd-recovery-stats/osd.2.log 2024-04-25T08:58:38.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:227: TEST_recovery_sizeup: check td/osd-recovery-stats 1.0 2 replicated 400 0 400 0 200 0 2024-04-25T08:58:38.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:58:38.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:58:38.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=2 2024-04-25T08:58:38.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=400 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=200 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.2.log 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:58:38.086 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:58:38.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[2,4,3,5]/[2,3]' 2024-04-25T08:58:38.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:58:38.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:58:38.110 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:58:38.110 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T08:58:38.110 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:58:38.111 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:58:38.111 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:58:38.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=400 2024-04-25T08:58:38.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 400 400 2024-04-25T08:58:38.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-04-25T08:58:38.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:58:38.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T08:58:38.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:58:38.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:58:38.115 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:58:38.115 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:58:38.115 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:58:38.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:58:38.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:58:38.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:58:38.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:58:38.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:58:38.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:58:38.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:58:38.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:58:38.126 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:58:38.126 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T08:58:38.127 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:58:38.127 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:58:38.127 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:58:38.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-04-25T08:58:38.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-04-25T08:58:38.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-04-25T08:58:38.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:58:38.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T08:58:38.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:58:38.130 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:58:38.130 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:58:38.130 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:58:38.130 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:58:38.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:58:38.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:58:38.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:58:38.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:58:38.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:58:38.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:58:38.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 200 ']' 2024-04-25T08:58:38.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 2' 2024-04-25T08:58:38.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:58:38.140 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:58:38.140 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T08:58:38.141 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-04-25T08:58:38.141 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-04-25T08:58:38.141 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:58:38.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=200 2024-04-25T08:58:38.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 200 200 2024-04-25T08:58:38.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:58:38.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:58:38.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:58:38.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:58:38.144 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:58:38.144 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-04-25T08:58:38.144 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-04-25T08:58:38.144 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:58:38.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-04-25T08:58:38.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-04-25T08:58:38.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:58:38.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:58:38.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:58:38.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:58:38.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:229: TEST_recovery_sizeup: delete_pool test 2024-04-25T08:58:38.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:58:38.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:58:38.668 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T08:58:38.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:230: TEST_recovery_sizeup: kill_daemons td/osd-recovery-stats 2024-04-25T08:58:38.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:58:38.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:58:38.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:58:38.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:58:38.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:58:44.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:58:44.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:58:44.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:58:44.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:58:44.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:58:44.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:58:44.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:58:44.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:58:44.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:58:44.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:58:44.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:58:44.015 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:58:44.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:58:44.016 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:58:44.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:58:44.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:58:44.018 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:58:44.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:58:44.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:58:44.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:58:44.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:58:44.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:58:44.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:58:44.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:58:44.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:58:44.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:44.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:58:44.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:58:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:58:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:58:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:58:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:58:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:58:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:58:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:58:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:58:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:58:44.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:58:44.069 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:58:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:58:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:58:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:58:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:58:44.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:58:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:58:44.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:58:44.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:58:44.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:58:44.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:58:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:58:44.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:58:44.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:58:44.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:58:44.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:58:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:58:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:58:44.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:58:44.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:44.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:58:44.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T08:58:44.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:58:44.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:58:44.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:58:44.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:58:44.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:44.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:58:44.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.277516 2024-04-25T08:58:44.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:58:44.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:58:44.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:58:44.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:58:44.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_undersized td/osd-recovery-stats 2024-04-25T08:58:44.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:306: TEST_recovery_undersized: local dir=td/osd-recovery-stats 2024-04-25T08:58:44.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:308: TEST_recovery_undersized: local osds=3 2024-04-25T08:58:44.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:309: TEST_recovery_undersized: run_mon td/osd-recovery-stats a 2024-04-25T08:58:44.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:58:44.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:58:44.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:58:44.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:58:44.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:58:44.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:58:44.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:58:44.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:58:44.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:58:44.556 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:58:44.556 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:44.556 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:58:44.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:58:44.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:58:44.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:58:44.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:58:44.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:58:44.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:58:44.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:58:44.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:58:44.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:58:44.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:58:44.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:58:44.795 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:44.795 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:44.796 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:58:44.796 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:58:44.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:58:44.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get fsid 2024-04-25T08:58:44.912 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:58:44.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:58:44.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:58:44.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:58:44.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:58:44.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:58:44.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:58:44.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:58:44.915 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:58:44.915 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:44.916 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:58:44.916 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.277516/ceph-mon.a.asok 2024-04-25T08:58:44.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:58:44.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.277516/ceph-mon.a.asok config get mon_host 2024-04-25T08:58:45.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:310: TEST_recovery_undersized: run_mgr td/osd-recovery-stats x 2024-04-25T08:58:45.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:58:45.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:58:45.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:58:45.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:58:45.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:58:45.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:58:45.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:58:45.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:58:45.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:58:45.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:58:45.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:45.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:58:45.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:58:45.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:58:45.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:58:45.986 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: expr 3 - 1 2024-04-25T08:58:45.986 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: seq 0 2 2024-04-25T08:58:45.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 0 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:58:45.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:58:45.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:58:45.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:58:45.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:58:45.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:58:45.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:58:45.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:58:45.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:58:45.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:58:45.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b1ec6a74-6c6b-4ab2-b515-98e866581068 2024-04-25T08:58:45.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b1ec6a74-6c6b-4ab2-b515-98e866581068' 2024-04-25T08:58:45.993 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 b1ec6a74-6c6b-4ab2-b515-98e866581068 2024-04-25T08:58:45.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:58:46.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGGypmTw1DARAADu48QpK8gPDFOSJqRQAuNg== 2024-04-25T08:58:46.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGGypmTw1DARAADu48QpK8gPDFOSJqRQAuNg=="}' 2024-04-25T08:58:46.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b1ec6a74-6c6b-4ab2-b515-98e866581068 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:58:46.311 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:58:46.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:58:46.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGGypmTw1DARAADu48QpK8gPDFOSJqRQAuNg== --osd-uuid b1ec6a74-6c6b-4ab2-b515-98e866581068 2024-04-25T08:58:46.355 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:46.366+0000 7f2d672bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:46.380 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:46.390+0000 7f2d672bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:46.381 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:46.390+0000 7f2d672bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:46.381 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:46.390+0000 7f2d672bf7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:58:46.382 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:46.390+0000 7f2d672bf7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:58:46.382 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:46.390+0000 7f2d672bf7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:58:46.419 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:46.430+0000 7f2d672bf7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:58:52.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:58:52.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:58:52.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:58:52.926 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T08:58:52.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:58:53.479 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T08:58:53.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:58:53.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:58:53.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:58:53.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:58:53.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:58:53.511 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:53.522+0000 7fa5c3ebd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:53.513 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:53.522+0000 7fa5c3ebd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:53.515 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:53.526+0000 7fa5c3ebd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:58:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:58:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:58:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:58:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:58:53.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:53.929 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:58:53.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:58:53.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:58:53.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:54.072 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:54.082+0000 7fa5c3ebd7c0 -1 Falling back to public interface 2024-04-25T08:58:54.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:55.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:55.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:55.359 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:58:55.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:58:55.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:55.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:58:55.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:56.732 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:58:56.742+0000 7fa5c3ebd7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:58:56.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:56.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:56.768 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:58:56.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:58:56.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:56.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:58:57.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:58.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:58.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:58.518 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:58:58.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:58:58.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:58.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:58:58.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:59.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:59.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:59.983 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:58:59.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:58:59.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:59.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:00.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:01.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:01.414 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:59:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:59:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:01.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:01.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:02.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:02.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:02.999 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:59:03.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:59:03.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:03.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:03.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:04.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:04.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:04.441 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:59:04.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:59:04.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:04.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:05.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:06.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:06.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:06.006 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:59:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:59:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:06.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:06.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:07.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:07.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:07.430 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:59:07.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:59:07.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:07.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:08.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:09.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:09.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:09.008 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:59:09.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:59:09.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:09.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:09.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:10.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:10.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:10.457 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:59:10.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:59:10.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:10.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:11.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:12.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:12.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:12.343 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:59:12.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:59:12.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:12.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:13.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:14.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:14.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:14.427 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:59:14.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:59:14.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:14.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:15.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:16.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:16.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:16.819 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:59:16.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:59:16.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:16.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:17.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:18.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:18.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:18.520 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:59:18.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:59:18.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:18.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:19.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:20.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:20.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:20.491 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:59:20.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:59:20.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:20.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:21.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:22.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:22.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:22.102 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:59:22.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:59:22.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:22.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:59:22.607 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/71855892,v1:127.0.0.1:6803/71855892] [v2:127.0.0.1:6804/71855892,v1:127.0.0.1:6805/71855892] exists,up b1ec6a74-6c6b-4ab2-b515-98e866581068 2024-04-25T08:59:22.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:59:22.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 1 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:59:22.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:59:22.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:59:22.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:59:22.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:59:22.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:59:22.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:59:22.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:59:22.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:59:22.612 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:59:22.612 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:59:22.612 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T08:59:22.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:59:22.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T08:59:22.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:59:22.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:59:22.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:59:22.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:59:22.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:59:22.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:59:22.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:59:22.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:59:22.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:59:22.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:59:22.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:59:22.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:59:22.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:59:22.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ae00254-2620-44de-993b-7311d37e8deb 2024-04-25T08:59:22.614 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 7ae00254-2620-44de-993b-7311d37e8deb 2024-04-25T08:59:22.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7ae00254-2620-44de-993b-7311d37e8deb' 2024-04-25T08:59:22.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:59:22.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBqGypmiOyIJhAA36sp/GyxnjTuzF7T08HpCw== 2024-04-25T08:59:22.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBqGypmiOyIJhAA36sp/GyxnjTuzF7T08HpCw=="}' 2024-04-25T08:59:22.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ae00254-2620-44de-993b-7311d37e8deb -i td/osd-recovery-stats/1/new.json 2024-04-25T08:59:23.253 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:59:23.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:59:23.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBqGypmiOyIJhAA36sp/GyxnjTuzF7T08HpCw== --osd-uuid 7ae00254-2620-44de-993b-7311d37e8deb 2024-04-25T08:59:23.304 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:23.314+0000 7fdf4fc657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:23.345 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:23.354+0000 7fdf4fc657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:23.346 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:23.358+0000 7fdf4fc657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:23.347 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:23.358+0000 7fdf4fc657c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:59:23.347 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:23.358+0000 7fdf4fc657c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:59:23.347 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:23.358+0000 7fdf4fc657c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:59:23.384 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:23.394+0000 7fdf4fc657c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:59:30.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:59:30.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:59:30.188 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T08:59:30.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:59:30.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:59:30.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:59:30.751 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T08:59:30.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:59:30.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:59:30.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:59:30.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:59:30.777 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:30.785+0000 7f4079ca97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:30.780 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:30.789+0000 7f4079ca97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:30.781 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:30.789+0000 7f4079ca97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:31.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:59:31.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:59:31.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:59:31.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:59:31.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:59:31.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:31.210 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T08:59:31.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:59:31.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:31.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:31.327 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:31.337+0000 7f4079ca97c0 -1 Falling back to public interface 2024-04-25T08:59:31.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:32.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:32.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:32.613 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T08:59:32.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:59:32.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:32.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:33.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:33.949 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T08:59:33.957+0000 7f4079ca97c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:59:34.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:34.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:34.028 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T08:59:34.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:59:34.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:34.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:34.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:35.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:35.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:35.856 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T08:59:35.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:59:35.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:35.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:36.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:37.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:37.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:37.316 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T08:59:37.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:59:37.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:37.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:37.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:38.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:38.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:38.898 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T08:59:38.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:59:38.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:38.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:39.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:40.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:40.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:40.482 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T08:59:40.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:59:40.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:40.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:40.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:41.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:41.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:41.994 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T08:59:41.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:59:41.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:41.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:42.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:43.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:43.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:43.466 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T08:59:43.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:59:43.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:43.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:44.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:45.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:45.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:45.030 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T08:59:45.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:59:45.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:45.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:45.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:46.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:46.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:46.572 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T08:59:46.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:59:46.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:46.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:47.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:48.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:48.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:48.059 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T08:59:48.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:59:48.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:48.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:48.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:49.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:49.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:49.723 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T08:59:49.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:59:49.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:49.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:50.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:51.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:51.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:51.547 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T08:59:51.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:59:51.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:51.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:52.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:53.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:53.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:53.295 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T08:59:53.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:59:53.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:53.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:54.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:55.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:55.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:55.378 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T08:59:55.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:59:55.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:55.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:56.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:57.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:57.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:57.328 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T08:59:57.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:59:57.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:57.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:59:58.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:59.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:59.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:59.115 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T08:59:59.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:59:59.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:59.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:00.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:01.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:01.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:01.009 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:00:01.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:00:01.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:01.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:00:01.416 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1599960685,v1:127.0.0.1:6811/1599960685] [v2:127.0.0.1:6812/1599960685,v1:127.0.0.1:6813/1599960685] exists,up 7ae00254-2620-44de-993b-7311d37e8deb 2024-04-25T09:00:01.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:00:01.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:00:01.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:00:01.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T09:00:01.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 2 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:00:01.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T09:00:01.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:00:01.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:00:01.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:00:01.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:00:01.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:00:01.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:00:01.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T09:00:01.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:00:01.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2d4095d3-8c47-4d2d-b93a-c08e74e05082 2024-04-25T09:00:01.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2d4095d3-8c47-4d2d-b93a-c08e74e05082' 2024-04-25T09:00:01.424 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 2d4095d3-8c47-4d2d-b93a-c08e74e05082 2024-04-25T09:00:01.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:00:01.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRGypmcw9GGxAA4SWfbAQE8dp+IztmlgonDA== 2024-04-25T09:00:01.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRGypmcw9GGxAA4SWfbAQE8dp+IztmlgonDA=="}' 2024-04-25T09:00:01.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2d4095d3-8c47-4d2d-b93a-c08e74e05082 -i td/osd-recovery-stats/2/new.json 2024-04-25T09:00:01.944 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:00:01.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T09:00:01.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRGypmcw9GGxAA4SWfbAQE8dp+IztmlgonDA== --osd-uuid 2d4095d3-8c47-4d2d-b93a-c08e74e05082 2024-04-25T09:00:01.997 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:02.005+0000 7f1b9ac0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:02.041 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:02.049+0000 7f1b9ac0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:02.042 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:02.053+0000 7f1b9ac0f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:02.043 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:02.053+0000 7f1b9ac0f7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T09:00:02.043 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:02.053+0000 7f1b9ac0f7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T09:00:02.043 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:02.053+0000 7f1b9ac0f7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T09:00:02.087 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:02.097+0000 7f1b9ac0f7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T09:00:09.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T09:00:09.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:00:09.087 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T09:00:09.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:00:09.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:00:09.646 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:00:09.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:00:09.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2c0973da-b645-4a51-b630-38425e271a1f --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.277516/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:00:09.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:00:09.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:00:09.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:00:09.679 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:09.689+0000 7fe1670af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:09.681 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:09.689+0000 7fe1670af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:09.682 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:09.693+0000 7fe1670af7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:00:10.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:00:10.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:00:10.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:00:10.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:00:10.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:00:10.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:10.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:00:10.052 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:00:10.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:10.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:10.235 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:10.245+0000 7fe1670af7c0 -1 Falling back to public interface 2024-04-25T09:00:10.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:11.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:11.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:11.484 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:00:11.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:00:11.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:11.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:11.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:12.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:12.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:12.896 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:00:12.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:00:12.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:12.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:12.941 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:00:12.948+0000 7fe1670af7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:00:13.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:14.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:14.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:14.365 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:00:14.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:00:14.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:14.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:14.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:15.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:15.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:15.810 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:00:15.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:00:15.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:15.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:16.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:17.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:17.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:17.258 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:00:17.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:00:17.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:17.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:17.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:18.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:18.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:18.814 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:00:18.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:00:18.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:18.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:19.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:20.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:20.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:20.374 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:00:20.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:00:20.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:20.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:20.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:21.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:21.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:21.809 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:00:21.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:00:21.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:21.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:22.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:23.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:23.280 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:00:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:00:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:23.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:23.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:24.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:24.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:24.840 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:00:24.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:00:24.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:24.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:25.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:26.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:26.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:26.407 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:00:26.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:00:26.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:26.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:27.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:28.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:28.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:28.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:00:28.049 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:00:28.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:28.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:28.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:29.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:29.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:29.796 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:00:29.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:00:29.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:29.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:30.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:31.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:31.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:31.719 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:00:31.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:00:31.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:31.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:32.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:33.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:33.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:33.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:00:33.532 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:00:33.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:33.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:34.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:35.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:35.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:35.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:00:35.153 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:00:35.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:35.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:36.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:37.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:37.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:37.231 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:00:37.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:00:37.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:37.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:38.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:39.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:39.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:39.042 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:00:39.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:00:39.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:39.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:39.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:00:40.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:00:40.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:00:40.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:00:40.528 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T09:00:40.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:00:40.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:00:40.940 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4099485129,v1:127.0.0.1:6819/4099485129] [v2:127.0.0.1:6820/4099485129,v1:127.0.0.1:6821/4099485129] exists,up 2d4095d3-8c47-4d2d-b93a-c08e74e05082 2024-04-25T09:00:40.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:00:40.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:00:40.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:00:40.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:316: TEST_recovery_undersized: create_pool test 1 1 2024-04-25T09:00:40.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T09:00:41.540 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T09:00:41.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:00:42.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:317: TEST_recovery_undersized: ceph osd pool set test size 1 --yes-i-really-mean-it 2024-04-25T09:00:43.773 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 1 2024-04-25T09:00:43.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:319: TEST_recovery_undersized: wait_for_clean 2024-04-25T09:00:43.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:00:43.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:00:43.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:00:43.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:00:43.798 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:00:43.798 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:00:43.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:00:43.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:00:43.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:00:43.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:00:43.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:00:43.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:00:43.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:00:43.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:00:43.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:00:44.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:00:44.446 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:00:44.446 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:00:44.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:00:44.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:44.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:00:44.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836503 2024-04-25T09:00:44.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836503 2024-04-25T09:00:44.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503' 2024-04-25T09:00:44.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:44.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:00:44.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-04-25T09:00:44.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-04-25T09:00:44.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672978' 2024-04-25T09:00:44.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:44.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:00:45.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509449 2024-04-25T09:00:45.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509449 2024-04-25T09:00:45.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836503 1-42949672978 2-64424509449' 2024-04-25T09:00:45.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:45.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836503 2024-04-25T09:00:45.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:45.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:00:45.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836503 2024-04-25T09:00:45.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:45.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836503 2024-04-25T09:00:45.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-04-25T09:00:45.098 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836503 2024-04-25T09:00:45.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:00:45.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836503 2024-04-25T09:00:45.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:00:46.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:00:46.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:00:46.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836503 2024-04-25T09:00:46.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:46.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-04-25T09:00:46.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:46.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:00:46.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-04-25T09:00:46.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:46.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-04-25T09:00:46.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-25T09:00:46.971 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672978 2024-04-25T09:00:46.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:00:47.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672979 -lt 42949672978 2024-04-25T09:00:47.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:47.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509449 2024-04-25T09:00:47.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:47.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:00:47.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509449 2024-04-25T09:00:47.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:47.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509449 2024-04-25T09:00:47.443 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509449 2024-04-25T09:00:47.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-04-25T09:00:47.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:00:47.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509449 2024-04-25T09:00:47.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:00:47.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:47.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:48.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:00:48.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:00:48.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:00:48.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:00:48.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:00:48.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:00:48.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:00:48.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:00:48.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:00:48.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:00:48.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:48.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:49.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:00:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:00:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:00:49.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: seq 1 200 2024-04-25T09:00:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:49.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj1 /dev/null 2024-04-25T09:00:49.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:49.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj2 /dev/null 2024-04-25T09:00:49.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:49.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj3 /dev/null 2024-04-25T09:00:49.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:49.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj4 /dev/null 2024-04-25T09:00:49.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:49.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj5 /dev/null 2024-04-25T09:00:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:49.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj6 /dev/null 2024-04-25T09:00:49.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:49.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj7 /dev/null 2024-04-25T09:00:50.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj8 /dev/null 2024-04-25T09:00:50.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj9 /dev/null 2024-04-25T09:00:50.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj10 /dev/null 2024-04-25T09:00:50.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj11 /dev/null 2024-04-25T09:00:50.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj12 /dev/null 2024-04-25T09:00:50.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj13 /dev/null 2024-04-25T09:00:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj14 /dev/null 2024-04-25T09:00:50.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj15 /dev/null 2024-04-25T09:00:50.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj16 /dev/null 2024-04-25T09:00:50.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj17 /dev/null 2024-04-25T09:00:50.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj18 /dev/null 2024-04-25T09:00:50.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:50.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj19 /dev/null 2024-04-25T09:00:51.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:51.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj20 /dev/null 2024-04-25T09:00:51.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:51.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj21 /dev/null 2024-04-25T09:00:51.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:51.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj22 /dev/null 2024-04-25T09:00:51.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:51.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj23 /dev/null 2024-04-25T09:00:51.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:51.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj24 /dev/null 2024-04-25T09:00:51.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:51.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj25 /dev/null 2024-04-25T09:00:51.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:51.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj26 /dev/null 2024-04-25T09:00:51.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:51.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj27 /dev/null 2024-04-25T09:00:51.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:51.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj28 /dev/null 2024-04-25T09:00:51.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:51.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj29 /dev/null 2024-04-25T09:00:51.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:51.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj30 /dev/null 2024-04-25T09:00:52.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj31 /dev/null 2024-04-25T09:00:52.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj32 /dev/null 2024-04-25T09:00:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj33 /dev/null 2024-04-25T09:00:52.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj34 /dev/null 2024-04-25T09:00:52.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj35 /dev/null 2024-04-25T09:00:52.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj36 /dev/null 2024-04-25T09:00:52.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj37 /dev/null 2024-04-25T09:00:52.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj38 /dev/null 2024-04-25T09:00:52.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj39 /dev/null 2024-04-25T09:00:52.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj40 /dev/null 2024-04-25T09:00:52.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj41 /dev/null 2024-04-25T09:00:52.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj42 /dev/null 2024-04-25T09:00:52.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:52.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj43 /dev/null 2024-04-25T09:00:53.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:53.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj44 /dev/null 2024-04-25T09:00:53.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:53.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj45 /dev/null 2024-04-25T09:00:53.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:53.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj46 /dev/null 2024-04-25T09:00:53.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:53.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj47 /dev/null 2024-04-25T09:00:53.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:53.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj48 /dev/null 2024-04-25T09:00:53.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:53.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj49 /dev/null 2024-04-25T09:00:53.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:53.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj50 /dev/null 2024-04-25T09:00:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:53.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj51 /dev/null 2024-04-25T09:00:53.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:53.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj52 /dev/null 2024-04-25T09:00:53.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:53.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj53 /dev/null 2024-04-25T09:00:53.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:53.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj54 /dev/null 2024-04-25T09:00:54.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:54.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj55 /dev/null 2024-04-25T09:00:54.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:54.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj56 /dev/null 2024-04-25T09:00:54.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:54.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj57 /dev/null 2024-04-25T09:00:54.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:54.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj58 /dev/null 2024-04-25T09:00:54.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:54.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj59 /dev/null 2024-04-25T09:00:54.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:54.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj60 /dev/null 2024-04-25T09:00:54.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:54.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj61 /dev/null 2024-04-25T09:00:54.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:54.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj62 /dev/null 2024-04-25T09:00:54.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:54.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj63 /dev/null 2024-04-25T09:00:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:54.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj64 /dev/null 2024-04-25T09:00:54.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:54.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj65 /dev/null 2024-04-25T09:00:55.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj66 /dev/null 2024-04-25T09:00:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj67 /dev/null 2024-04-25T09:00:55.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj68 /dev/null 2024-04-25T09:00:55.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj69 /dev/null 2024-04-25T09:00:55.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj70 /dev/null 2024-04-25T09:00:55.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj71 /dev/null 2024-04-25T09:00:55.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj72 /dev/null 2024-04-25T09:00:55.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj73 /dev/null 2024-04-25T09:00:55.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj74 /dev/null 2024-04-25T09:00:55.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj75 /dev/null 2024-04-25T09:00:55.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj76 /dev/null 2024-04-25T09:00:55.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:55.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj77 /dev/null 2024-04-25T09:00:56.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:56.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj78 /dev/null 2024-04-25T09:00:56.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:56.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj79 /dev/null 2024-04-25T09:00:56.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:56.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj80 /dev/null 2024-04-25T09:00:56.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:56.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj81 /dev/null 2024-04-25T09:00:56.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:56.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj82 /dev/null 2024-04-25T09:00:56.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:56.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj83 /dev/null 2024-04-25T09:00:56.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:56.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj84 /dev/null 2024-04-25T09:00:56.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:56.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj85 /dev/null 2024-04-25T09:00:56.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:56.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj86 /dev/null 2024-04-25T09:00:56.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:56.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj87 /dev/null 2024-04-25T09:00:56.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:56.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj88 /dev/null 2024-04-25T09:00:57.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:57.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj89 /dev/null 2024-04-25T09:00:57.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:57.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj90 /dev/null 2024-04-25T09:00:57.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:57.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj91 /dev/null 2024-04-25T09:00:57.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:57.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj92 /dev/null 2024-04-25T09:00:57.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:57.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj93 /dev/null 2024-04-25T09:00:57.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:57.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj94 /dev/null 2024-04-25T09:00:57.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:57.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj95 /dev/null 2024-04-25T09:00:57.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:57.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj96 /dev/null 2024-04-25T09:00:57.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:57.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj97 /dev/null 2024-04-25T09:00:57.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:57.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj98 /dev/null 2024-04-25T09:00:57.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:57.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj99 /dev/null 2024-04-25T09:00:58.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj100 /dev/null 2024-04-25T09:00:58.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj101 /dev/null 2024-04-25T09:00:58.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj102 /dev/null 2024-04-25T09:00:58.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj103 /dev/null 2024-04-25T09:00:58.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj104 /dev/null 2024-04-25T09:00:58.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj105 /dev/null 2024-04-25T09:00:58.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj106 /dev/null 2024-04-25T09:00:58.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj107 /dev/null 2024-04-25T09:00:58.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj108 /dev/null 2024-04-25T09:00:58.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj109 /dev/null 2024-04-25T09:00:58.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj110 /dev/null 2024-04-25T09:00:58.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:58.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj111 /dev/null 2024-04-25T09:00:59.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:59.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj112 /dev/null 2024-04-25T09:00:59.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:59.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj113 /dev/null 2024-04-25T09:00:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:59.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj114 /dev/null 2024-04-25T09:00:59.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:59.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj115 /dev/null 2024-04-25T09:00:59.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:59.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj116 /dev/null 2024-04-25T09:00:59.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:59.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj117 /dev/null 2024-04-25T09:00:59.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:59.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj118 /dev/null 2024-04-25T09:00:59.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:59.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj119 /dev/null 2024-04-25T09:00:59.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:59.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj120 /dev/null 2024-04-25T09:00:59.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:59.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj121 /dev/null 2024-04-25T09:00:59.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:00:59.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj122 /dev/null 2024-04-25T09:01:00.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:00.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj123 /dev/null 2024-04-25T09:01:00.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:00.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj124 /dev/null 2024-04-25T09:01:00.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:00.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj125 /dev/null 2024-04-25T09:01:00.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:00.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj126 /dev/null 2024-04-25T09:01:00.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:00.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj127 /dev/null 2024-04-25T09:01:00.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:00.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj128 /dev/null 2024-04-25T09:01:00.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:00.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj129 /dev/null 2024-04-25T09:01:00.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:00.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj130 /dev/null 2024-04-25T09:01:00.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:00.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj131 /dev/null 2024-04-25T09:01:00.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:00.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj132 /dev/null 2024-04-25T09:01:00.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:00.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj133 /dev/null 2024-04-25T09:01:01.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj134 /dev/null 2024-04-25T09:01:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj135 /dev/null 2024-04-25T09:01:01.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj136 /dev/null 2024-04-25T09:01:01.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj137 /dev/null 2024-04-25T09:01:01.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj138 /dev/null 2024-04-25T09:01:01.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj139 /dev/null 2024-04-25T09:01:01.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj140 /dev/null 2024-04-25T09:01:01.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj141 /dev/null 2024-04-25T09:01:01.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj142 /dev/null 2024-04-25T09:01:01.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj143 /dev/null 2024-04-25T09:01:01.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj144 /dev/null 2024-04-25T09:01:01.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:01.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj145 /dev/null 2024-04-25T09:01:02.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:02.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj146 /dev/null 2024-04-25T09:01:02.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:02.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj147 /dev/null 2024-04-25T09:01:02.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:02.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj148 /dev/null 2024-04-25T09:01:02.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:02.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj149 /dev/null 2024-04-25T09:01:02.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:02.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj150 /dev/null 2024-04-25T09:01:02.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:02.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj151 /dev/null 2024-04-25T09:01:02.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:02.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj152 /dev/null 2024-04-25T09:01:02.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:02.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj153 /dev/null 2024-04-25T09:01:02.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:02.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj154 /dev/null 2024-04-25T09:01:02.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:02.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj155 /dev/null 2024-04-25T09:01:02.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:02.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj156 /dev/null 2024-04-25T09:01:03.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:03.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj157 /dev/null 2024-04-25T09:01:03.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:03.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj158 /dev/null 2024-04-25T09:01:03.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:03.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj159 /dev/null 2024-04-25T09:01:03.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:03.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj160 /dev/null 2024-04-25T09:01:03.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:03.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj161 /dev/null 2024-04-25T09:01:03.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:03.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj162 /dev/null 2024-04-25T09:01:03.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:03.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj163 /dev/null 2024-04-25T09:01:03.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:03.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj164 /dev/null 2024-04-25T09:01:03.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:03.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj165 /dev/null 2024-04-25T09:01:03.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:03.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj166 /dev/null 2024-04-25T09:01:03.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:03.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj167 /dev/null 2024-04-25T09:01:04.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:04.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj168 /dev/null 2024-04-25T09:01:04.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:04.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj169 /dev/null 2024-04-25T09:01:04.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:04.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj170 /dev/null 2024-04-25T09:01:04.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:04.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj171 /dev/null 2024-04-25T09:01:04.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:04.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj172 /dev/null 2024-04-25T09:01:04.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:04.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj173 /dev/null 2024-04-25T09:01:04.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:04.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj174 /dev/null 2024-04-25T09:01:04.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:04.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj175 /dev/null 2024-04-25T09:01:04.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:04.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj176 /dev/null 2024-04-25T09:01:04.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:04.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj177 /dev/null 2024-04-25T09:01:04.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:04.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj178 /dev/null 2024-04-25T09:01:05.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:05.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj179 /dev/null 2024-04-25T09:01:05.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:05.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj180 /dev/null 2024-04-25T09:01:05.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:05.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj181 /dev/null 2024-04-25T09:01:05.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:05.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj182 /dev/null 2024-04-25T09:01:05.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:05.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj183 /dev/null 2024-04-25T09:01:05.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:05.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj184 /dev/null 2024-04-25T09:01:05.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:05.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj185 /dev/null 2024-04-25T09:01:05.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:05.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj186 /dev/null 2024-04-25T09:01:05.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:05.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj187 /dev/null 2024-04-25T09:01:05.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:05.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj188 /dev/null 2024-04-25T09:01:05.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:05.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj189 /dev/null 2024-04-25T09:01:06.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:06.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj190 /dev/null 2024-04-25T09:01:06.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:06.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj191 /dev/null 2024-04-25T09:01:06.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:06.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj192 /dev/null 2024-04-25T09:01:06.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:06.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj193 /dev/null 2024-04-25T09:01:06.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:06.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj194 /dev/null 2024-04-25T09:01:06.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:06.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj195 /dev/null 2024-04-25T09:01:06.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:06.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj196 /dev/null 2024-04-25T09:01:06.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:06.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj197 /dev/null 2024-04-25T09:01:06.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:06.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj198 /dev/null 2024-04-25T09:01:06.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:06.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj199 /dev/null 2024-04-25T09:01:06.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:01:06.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj200 /dev/null 2024-04-25T09:01:06.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: get_primary test obj1 2024-04-25T09:01:06.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:01:06.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:01:06.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:01:06.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:01:07.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: local primary=1 2024-04-25T09:01:07.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: get_pg test obj1 2024-04-25T09:01:07.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T09:01:07.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T09:01:07.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T09:01:07.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T09:01:08.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: local PG=1.0 2024-04-25T09:01:08.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:329: TEST_recovery_undersized: ceph osd set norecover 2024-04-25T09:01:08.830 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is set 2024-04-25T09:01:08.851 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: expr 3 - 1 2024-04-25T09:01:08.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: seq 0 2 2024-04-25T09:01:08.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T09:01:08.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:333: TEST_recovery_undersized: '[' 0 = 1 ']' 2024-04-25T09:01:08.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:337: TEST_recovery_undersized: ceph osd out osd.0 2024-04-25T09:01:09.363 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. 2024-04-25T09:01:09.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:338: TEST_recovery_undersized: break 2024-04-25T09:01:09.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:340: TEST_recovery_undersized: ceph osd pool set test size 4 2024-04-25T09:01:10.018 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 4 2024-04-25T09:01:10.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:341: TEST_recovery_undersized: ceph osd unset norecover 2024-04-25T09:01:10.862 INFO:tasks.workunit.client.0.smithi102.stderr:norecover is unset 2024-04-25T09:01:10.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: get_primary test obj1 2024-04-25T09:01:10.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:01:10.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:01:10.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:01:10.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:01:11.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T09:01:11.552 INFO:tasks.workunit.client.0.smithi102.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T09:01:11.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:344: TEST_recovery_undersized: sleep 10 2024-04-25T09:01:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:345: TEST_recovery_undersized: flush_pg_stats 2024-04-25T09:01:21.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:01:21.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:01:21.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:01:21.998 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:01:21.998 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:01:21.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:01:21.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:01:21.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:01:22.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836512 2024-04-25T09:01:22.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836512 2024-04-25T09:01:22.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512' 2024-04-25T09:01:22.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:01:22.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:01:22.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672988 2024-04-25T09:01:22.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672988 2024-04-25T09:01:22.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672988' 2024-04-25T09:01:22.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:01:22.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:01:22.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509459 2024-04-25T09:01:22.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509459 2024-04-25T09:01:22.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672988 2-64424509459' 2024-04-25T09:01:22.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:01:22.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836512 2024-04-25T09:01:22.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:01:22.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:01:22.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836512 2024-04-25T09:01:22.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:01:22.541 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836512 2024-04-25T09:01:22.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836512 2024-04-25T09:01:22.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-04-25T09:01:22.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:01:23.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836512 -lt 21474836512 2024-04-25T09:01:23.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:01:23.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672988 2024-04-25T09:01:23.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:01:23.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:01:23.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672988 2024-04-25T09:01:23.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:01:23.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672988 2024-04-25T09:01:23.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-04-25T09:01:23.010 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672988 2024-04-25T09:01:23.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:01:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672988 -lt 42949672988 2024-04-25T09:01:23.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:01:23.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509459 2024-04-25T09:01:23.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:01:23.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:01:23.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509459 2024-04-25T09:01:23.456 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:01:23.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509459 2024-04-25T09:01:23.457 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509459 2024-04-25T09:01:23.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-04-25T09:01:23.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:01:23.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509459 2024-04-25T09:01:23.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:01:24.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:01:24.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:01:25.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509459 -lt 64424509459 2024-04-25T09:01:25.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: seq 1 300 2024-04-25T09:01:25.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 300) 2024-04-25T09:01:25.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2024-04-25T09:01:25.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2024-04-25T09:01:25.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2024-04-25T09:01:25.727 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T09:01:25.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:356: TEST_recovery_undersized: '[' 1 = 300 ']' 2024-04-25T09:01:25.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:361: TEST_recovery_undersized: sleep 1 2024-04-25T09:01:26.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 300) 2024-04-25T09:01:26.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2024-04-25T09:01:26.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2024-04-25T09:01:26.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2024-04-25T09:01:27.131 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T09:01:27.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:354: TEST_recovery_undersized: break 2024-04-25T09:01:27.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: get_primary test obj1 2024-04-25T09:01:27.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:01:27.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:01:27.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:01:27.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:01:27.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: primary=1 2024-04-25T09:01:27.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:366: TEST_recovery_undersized: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T09:01:27.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: expr 200 '*' 3 2024-04-25T09:01:27.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: local first_degraded=600 2024-04-25T09:01:27.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: expr 200 '*' 2 2024-04-25T09:01:27.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: local last_degraded=400 2024-04-25T09:01:27.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:370: TEST_recovery_undersized: check td/osd-recovery-stats 1.0 1 replicated 600 400 0 0 2024-04-25T09:01:27.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T09:01:27.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T09:01:27.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T09:01:27.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T09:01:27.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=600 2024-04-25T09:01:27.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=400 2024-04-25T09:01:27.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T09:01:27.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T09:01:27.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T09:01:27.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T09:01:27.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T09:01:27.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T09:01:27.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T09:01:27.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T09:01:27.571 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T09:01:27.571 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T09:01:27.571 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T09:01:27.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2]' 2024-04-25T09:01:27.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T09:01:27.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:01:27.584 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T09:01:27.585 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T09:01:27.585 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2] ' 2024-04-25T09:01:27.585 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T09:01:27.585 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:01:27.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=600 2024-04-25T09:01:27.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 600 600 2024-04-25T09:01:27.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=600 2024-04-25T09:01:27.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T09:01:27.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=600 2024-04-25T09:01:27.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T09:01:27.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:01:27.591 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T09:01:27.591 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T09:01:27.591 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:01:27.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=400 2024-04-25T09:01:27.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 400 400 2024-04-25T09:01:27.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=400 2024-04-25T09:01:27.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T09:01:27.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=400 2024-04-25T09:01:27.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T09:01:27.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T09:01:27.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:01:27.601 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T09:01:27.601 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2] ' 2024-04-25T09:01:27.601 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T09:01:27.601 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T09:01:27.601 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:01:27.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T09:01:27.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T09:01:27.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T09:01:27.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T09:01:27.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T09:01:27.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T09:01:27.606 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:01:27.607 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T09:01:27.607 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T09:01:27.607 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:01:27.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T09:01:27.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T09:01:27.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T09:01:27.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T09:01:27.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T09:01:27.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T09:01:27.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T09:01:27.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:372: TEST_recovery_undersized: delete_pool test 2024-04-25T09:01:27.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T09:01:27.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T09:01:28.125 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T09:01:28.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:373: TEST_recovery_undersized: kill_daemons td/osd-recovery-stats 2024-04-25T09:01:28.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:01:28.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:01:28.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:01:28.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:01:28.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:01:33.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:01:33.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T09:01:33.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T09:01:33.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:01:33.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T09:01:33.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:01:33.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:01:33.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:01:33.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:01:33.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:01:33.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:01:33.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:01:33.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:01:33.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:01:33.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:01:33.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:01:33.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:01:33.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:01:33.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:01:33.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:01:33.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:01:33.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:01:33.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:01:33.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T09:01:33.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:01:33.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:33.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T09:01:33.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-stats 0 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:01:33.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:01:33.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:01:33.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:01:33.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:01:33.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:01:33.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:01:33.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:01:33.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:01:33.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:01:33.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:01:33.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:01:33.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:01:33.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:01:33.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:01:33.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T09:01:33.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:01:33.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:33.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.277516 2024-04-25T09:01:33.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.277516 2024-04-25T09:01:33.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:01:33.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:01:33.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:01:33.512 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:01:33.512 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:01:33.565 INFO:tasks.workunit:Running workunit osd/osd-rep-recov-eio.sh... 2024-04-25T09:01:33.565 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-rep-recov-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh 2024-04-25T09:01:33.617 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:01:33.623 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:01:33.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:01:33.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:01:33.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:01:33.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:01:33.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:01:33.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:01:33.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:01:33.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:01:33.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-rep-recov-eio.sh 2024-04-25T09:01:33.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:25: run: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:33.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:26: run: shift 2024-04-25T09:01:33.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: export CEPH_MON=127.0.0.1:7140 2024-04-25T09:01:33.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: CEPH_MON=127.0.0.1:7140 2024-04-25T09:01:33.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:29: run: export CEPH_ARGS 2024-04-25T09:01:33.624 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: uuidgen 2024-04-25T09:01:33.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: CEPH_ARGS+='--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none ' 2024-04-25T09:01:33.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:31: run: CEPH_ARGS+='--mon-host=127.0.0.1:7140 ' 2024-04-25T09:01:33.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: set 2024-04-25T09:01:33.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: local 'funcs=TEST_rados_get_with_eio 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_rados_repair_warning 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_rep_backfill_unfound 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_rep_recovery_unfound' 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:01:33.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:01:33.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:01:33.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:01:33.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:01:33.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:01:33.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:01:33.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:01:33.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:01:33.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:01:33.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:01:33.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:01:33.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:01:33.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:01:33.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:01:33.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:01:33.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:01:33.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:01:33.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:01:33.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:01:33.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:01:33.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:33.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:01:33.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.388535 2024-04-25T09:01:33.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:01:33.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:01:33.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T09:01:33.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:01:33.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:33.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:01:33.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.388535 2024-04-25T09:01:33.644 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:01:33.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:01:33.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:01:33.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:01:33.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T09:01:33.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T09:01:33.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:33.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:01:33.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:01:33.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:01:33.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T09:01:33.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T09:01:34.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:01:34.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:01:34.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:01:34.142 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:01:34.142 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:34.142 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:01:34.143 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:01:34.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T09:01:34.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:01:34.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:01:34.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:01:34.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:01:34.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:01:34.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:01:34.379 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:01:34.379 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:01:34.379 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:01:34.379 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:01:34.379 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:34.379 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:01:34.379 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-mon.a.asok 2024-04-25T09:01:34.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:01:34.380 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.388535/ceph-mon.a.asok config get fsid 2024-04-25T09:01:34.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:01:34.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:01:34.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:01:34.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:01:34.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:01:34.496 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:01:34.496 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:01:34.496 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:01:34.496 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:01:34.496 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:34.497 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:01:34.497 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-mon.a.asok 2024-04-25T09:01:34.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:01:34.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.388535/ceph-mon.a.asok config get mon_host 2024-04-25T09:01:34.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T09:01:34.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:34.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:01:34.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:01:34.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:01:34.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T09:01:34.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:01:35.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:01:35.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:01:35.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:01:35.535 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:01:35.535 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:35.535 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:01:35.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:01:35.536 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:01:35.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:01:35.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T09:01:35.881 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T09:01:35.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_get_with_eio td/osd-rep-recov-eio.sh 2024-04-25T09:01:35.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:168: TEST_rados_get_with_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:35.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:170: TEST_rados_get_with_eio: setup_osds 4 2024-04-25T09:01:35.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-04-25T09:01:35.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T09:01:35.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T09:01:35.899 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-04-25T09:01:35.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-04-25T09:01:35.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:01:35.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:01:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:01:35.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:01:35.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:01:35.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:01:35.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:01:35.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:01:35.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:01:35.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:01:35.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c68c3860-5dc2-4870-ba5d-ee8504bed474 2024-04-25T09:01:35.908 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 c68c3860-5dc2-4870-ba5d-ee8504bed474 2024-04-25T09:01:35.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c68c3860-5dc2-4870-ba5d-ee8504bed474' 2024-04-25T09:01:35.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:01:35.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvGypmML0GOBAA/WpELAqcobRIirwwSPsQCA== 2024-04-25T09:01:35.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvGypmML0GOBAA/WpELAqcobRIirwwSPsQCA=="}' 2024-04-25T09:01:35.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c68c3860-5dc2-4870-ba5d-ee8504bed474 -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:01:36.216 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:01:36.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:01:36.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvGypmML0GOBAA/WpELAqcobRIirwwSPsQCA== --osd-uuid c68c3860-5dc2-4870-ba5d-ee8504bed474 2024-04-25T09:01:36.267 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:36.279+0000 7fb6c89b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:36.284 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:36.295+0000 7fb6c89b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:36.286 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:36.295+0000 7fb6c89b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:36.286 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:36.299+0000 7fb6c89b77c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:01:36.286 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:36.299+0000 7fb6c89b77c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:01:36.286 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:36.299+0000 7fb6c89b77c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:01:36.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:36.327+0000 7fb6c89b77c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T09:01:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T09:01:42.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:01:42.858 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:01:42.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:01:42.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:01:43.400 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:01:43.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:01:43.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:01:43.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:01:43.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:01:43.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:01:43.430 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:43.439+0000 7f7578d357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:43.432 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:43.443+0000 7f7578d357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:43.433 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:43.443+0000 7f7578d357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:43.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:01:43.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:01:43.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:01:43.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:01:43.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:01:43.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:43.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:01:43.816 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:01:43.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:43.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:43.988 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:43.999+0000 7f7578d357c0 -1 Falling back to public interface 2024-04-25T09:01:44.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:45.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:45.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:45.247 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:01:45.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:01:45.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:45.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:45.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:46.598 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:01:46.610+0000 7f7578d357c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:01:46.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:46.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:46.698 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:01:46.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:01:46.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:46.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:47.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:48.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:48.324 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:01:48.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:48.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:01:48.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:48.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:48.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:49.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:49.754 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:01:49.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:49.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:01:49.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:49.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:50.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:51.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:51.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:51.188 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:01:51.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:01:51.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:51.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:51.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:52.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:52.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:52.792 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:01:52.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:01:52.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:52.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:53.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:54.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:54.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:54.231 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:01:54.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:01:54.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:54.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:54.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:55.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:55.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:55.797 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:01:55.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:01:55.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:55.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:56.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:57.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:57.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:57.216 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:01:57.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:01:57.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:57.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:57.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:58.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:58.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:58.798 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:01:58.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:01:58.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:58.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:59.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:00.239 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:02:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:02:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:01.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:02.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:02.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:02.004 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:02:02.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:02:02.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:02.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:02.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:03.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:03.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:03.677 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:02:03.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:02:03.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:03.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:04.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:05.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:05.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:05.437 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:02:05.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:02:05.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:05.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:06.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:07.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:07.267 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:02:07.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:07.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:02:07.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:07.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:08.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:09.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:09.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:09.210 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:02:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:02:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:10.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:11.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:11.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:11.322 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:02:11.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:02:11.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:11.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:11.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:12.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:12.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:12.881 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:02:12.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:02:12.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:12.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:02:13.345 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/468211697,v1:127.0.0.1:6803/468211697] [v2:127.0.0.1:6804/468211697,v1:127.0.0.1:6805/468211697] exists,up c68c3860-5dc2-4870-ba5d-ee8504bed474 2024-04-25T09:02:13.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:02:13.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:02:13.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:02:13.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:02:13.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T09:02:13.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:02:13.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:02:13.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:02:13.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:02:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T09:02:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:02:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:02:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:02:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:02:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T09:02:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T09:02:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:02:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:02:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:02:13.350 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:02:13.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:02:13.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:02:13.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:02:13.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:02:13.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:02:13.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:02:13.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T09:02:13.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:02:13.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a550677e-0511-4344-ac39-d294b38ab227 2024-04-25T09:02:13.353 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 a550677e-0511-4344-ac39-d294b38ab227 2024-04-25T09:02:13.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a550677e-0511-4344-ac39-d294b38ab227' 2024-04-25T09:02:13.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:02:13.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAVHCpmQ6XMFhAAlLatL2O8j/eX+jrme8VFtA== 2024-04-25T09:02:13.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAVHCpmQ6XMFhAAlLatL2O8j/eX+jrme8VFtA=="}' 2024-04-25T09:02:13.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a550677e-0511-4344-ac39-d294b38ab227 -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:02:13.872 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:02:13.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:02:13.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAVHCpmQ6XMFhAAlLatL2O8j/eX+jrme8VFtA== --osd-uuid a550677e-0511-4344-ac39-d294b38ab227 2024-04-25T09:02:13.920 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:13.930+0000 7fcd2fa557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:13.942 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:13.950+0000 7fcd2fa557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:13.943 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:13.954+0000 7fcd2fa557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:13.943 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:13.954+0000 7fcd2fa557c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:02:13.943 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:13.954+0000 7fcd2fa557c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:02:13.943 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:13.954+0000 7fcd2fa557c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:02:13.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:13.990+0000 7fcd2fa557c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T09:02:20.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T09:02:20.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:02:20.986 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:02:20.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:02:20.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:02:21.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:02:21.530 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:02:21.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:02:21.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:02:21.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:02:21.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:02:21.553 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:21.562+0000 7f52cf94a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:21.590 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:21.602+0000 7f52cf94a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:21.592 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:21.602+0000 7f52cf94a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:21.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:02:21.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:02:21.960 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:02:21.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:02:21.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:02:21.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:02:21.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:21.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:02:21.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:21.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:22.190 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:22.198+0000 7f52cf94a7c0 -1 Falling back to public interface 2024-04-25T09:02:22.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:23.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:23.430 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:02:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:23.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:02:23.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:23.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:23.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:24.801 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:24.810+0000 7f52cf94a7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:02:24.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:24.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:24.859 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:02:24.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:02:24.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:24.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:25.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:26.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:26.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:26.396 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:02:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:02:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:26.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:26.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:27.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:27.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:27.927 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:02:27.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:02:27.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:27.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:28.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:29.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:29.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:29.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:02:29.415 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:02:29.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:29.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:29.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:30.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:30.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:30.875 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:02:30.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:02:30.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:30.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:31.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:32.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:32.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:32.306 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:02:32.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:02:32.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:32.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:32.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:33.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:33.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:33.887 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:02:33.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:02:33.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:33.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:34.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:35.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:35.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:35.399 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:02:35.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:02:35.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:35.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:35.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:36.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:36.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:36.944 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:02:36.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:02:36.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:36.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:37.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:38.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:38.528 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:02:38.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:38.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:02:38.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:38.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:39.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:40.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:40.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:40.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:02:40.252 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:02:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:40.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:40.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:41.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:41.706 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:02:41.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:41.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:42.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:43.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:43.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:43.339 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:02:43.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:02:43.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:43.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:44.159 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:45.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:45.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:45.161 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:02:45.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:02:45.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:45.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:45.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:46.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:46.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:46.857 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:02:46.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:02:46.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:46.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:47.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:48.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:48.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:48.543 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:02:48.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:02:48.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:48.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:49.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:50.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:50.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:50.039 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:02:50.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:02:50.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:50.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:50.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:51.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:51.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:51.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:02:51.815 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T09:02:51.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:51.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:52.388 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1806503951,v1:127.0.0.1:6811/1806503951] [v2:127.0.0.1:6812/1806503951,v1:127.0.0.1:6813/1806503951] exists,up a550677e-0511-4344-ac39-d294b38ab227 2024-04-25T09:02:52.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:02:52.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:02:52.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:02:52.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:02:52.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:02:52.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:02:52.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:02:52.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:02:52.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:02:52.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:02:52.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:02:52.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:02:52.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:02:52.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:02:52.394 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 7ea6236a-4be7-4a6b-98b9-d8ecb4096f02 2024-04-25T09:02:52.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7ea6236a-4be7-4a6b-98b9-d8ecb4096f02 2024-04-25T09:02:52.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7ea6236a-4be7-4a6b-98b9-d8ecb4096f02' 2024-04-25T09:02:52.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:02:52.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8HCpme0F5GRAAyOlj8Lykfyy2ZzcF+Gma6A== 2024-04-25T09:02:52.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8HCpme0F5GRAAyOlj8Lykfyy2ZzcF+Gma6A=="}' 2024-04-25T09:02:52.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7ea6236a-4be7-4a6b-98b9-d8ecb4096f02 -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:02:53.652 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:02:53.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:02:53.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8HCpme0F5GRAAyOlj8Lykfyy2ZzcF+Gma6A== --osd-uuid 7ea6236a-4be7-4a6b-98b9-d8ecb4096f02 2024-04-25T09:02:53.707 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:53.717+0000 7fc069ce97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:53.725 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:53.733+0000 7fc069ce97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:53.726 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:53.737+0000 7fc069ce97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:53.727 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:53.737+0000 7fc069ce97c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:02:53.727 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:53.737+0000 7fc069ce97c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:02:53.727 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:53.737+0000 7fc069ce97c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:02:53.849 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:02:53.861+0000 7fc069ce97c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:03:01.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:03:01.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:03:01.031 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T09:03:01.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:03:01.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:03:01.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:03:01.585 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:03:01.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:03:01.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:03:01.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:03:01.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:03:01.614 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:01.621+0000 7f3d46c3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:01.616 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:01.625+0000 7f3d46c3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:01.618 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:01.625+0000 7f3d46c3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:01.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:03:01.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:03:01.974 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:03:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:03:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:03:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:03:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:03:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:02.172 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:02.181+0000 7f3d46c3e7c0 -1 Falling back to public interface 2024-04-25T09:03:02.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:03.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:03.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:03.403 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:03:03.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:03:03.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:03.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:03.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:04.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:04.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:04.820 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:03:04.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:03:04.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:04.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:04.864 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:04.873+0000 7f3d46c3e7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:03:05.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:06.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:06.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:06.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:03:06.453 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:03:06.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:06.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:07.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:08.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:08.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:08.087 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:03:08.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:03:08.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:08.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:08.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:09.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:09.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:09.723 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:03:09.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:03:09.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:09.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:10.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:11.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:11.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:11.183 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:03:11.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:03:11.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:11.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:11.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:12.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:12.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:12.606 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:03:12.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:03:12.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:12.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:13.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:14.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:14.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:14.231 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:03:14.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:03:14.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:14.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:14.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:15.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:15.789 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:03:15.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:03:15.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:15.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:16.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:17.194 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:03:17.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:03:17.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:17.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:17.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:18.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:18.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:18.600 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:03:18.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:03:18.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:18.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:19.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:20.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:20.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:20.200 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:03:20.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:03:20.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:20.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:21.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:22.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:22.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:22.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:03:22.218 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:03:22.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:22.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:22.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:23.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:23.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:23.894 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:03:23.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:03:23.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:23.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:24.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:25.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:25.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:25.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:03:25.635 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:03:25.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:25.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:26.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:27.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:27.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:27.340 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:03:27.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:03:27.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:27.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:28.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:29.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:29.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:29.235 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:03:29.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:03:29.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:29.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:30.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:31.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:31.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:31.192 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:03:31.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:03:31.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:31.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:32.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:33.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:33.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:33.018 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T09:03:33.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:03:33.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:33.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:33.793 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2777005445,v1:127.0.0.1:6819/2777005445] [v2:127.0.0.1:6820/2777005445,v1:127.0.0.1:6821/2777005445] exists,up 7ea6236a-4be7-4a6b-98b9-d8ecb4096f02 2024-04-25T09:03:33.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:03:33.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:03:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:03:33.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-04-25T09:03:33.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:03:33.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=293838be-5a59-490f-9336-3c6bca7896d9 2024-04-25T09:03:33.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 293838be-5a59-490f-9336-3c6bca7896d9' 2024-04-25T09:03:33.801 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 293838be-5a59-490f-9336-3c6bca7896d9 2024-04-25T09:03:33.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:03:33.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlHCpmfZZtMRAAjYTPKayivUn4zko30AZbxg== 2024-04-25T09:03:33.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlHCpmfZZtMRAAjYTPKayivUn4zko30AZbxg=="}' 2024-04-25T09:03:33.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 293838be-5a59-490f-9336-3c6bca7896d9 -i td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T09:03:34.719 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:03:34.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T09:03:34.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlHCpmfZZtMRAAjYTPKayivUn4zko30AZbxg== --osd-uuid 293838be-5a59-490f-9336-3c6bca7896d9 2024-04-25T09:03:34.767 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:34.776+0000 7f337ad667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:34.769 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:34.776+0000 7f337ad667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:34.771 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:34.780+0000 7f337ad667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:34.771 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:34.780+0000 7f337ad667c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:03:34.771 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:34.780+0000 7f337ad667c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:03:34.771 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:34.780+0000 7f337ad667c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:03:34.928 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:34.936+0000 7f337ad667c0 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-04-25T09:03:41.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-04-25T09:03:41.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:03:41.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T09:03:41.903 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T09:03:41.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:03:42.468 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T09:03:42.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T09:03:42.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:03:42.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:03:42.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:03:42.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:03:42.497 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:42.504+0000 7fbcfae0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:42.499 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:42.508+0000 7fbcfae0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:42.500 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:42.508+0000 7fbcfae0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:42.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T09:03:42.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:03:42.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T09:03:42.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:03:42.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:03:42.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:42.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:03:42.947 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:03:42.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:42.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:43.047 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:43.056+0000 7fbcfae0e7c0 -1 Falling back to public interface 2024-04-25T09:03:43.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:44.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:44.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:44.375 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:03:44.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:03:44.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:44.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:44.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:45.650 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:03:45.656+0000 7fbcfae0e7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T09:03:45.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:45.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:45.799 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:03:45.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:03:45.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:45.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:46.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:47.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:47.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:47.602 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:03:47.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:03:47.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:47.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:48.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:49.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:49.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:49.032 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:03:49.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:03:49.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:49.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:49.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:50.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:50.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:50.695 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:03:50.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:03:50.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:50.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:51.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:52.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:52.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:52.242 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:03:52.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:03:52.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:52.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:52.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:53.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:53.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:53.766 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:03:53.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:03:53.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:53.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:54.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:55.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:55.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:55.210 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:03:55.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:03:55.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:55.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:55.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:56.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:56.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:56.751 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:03:56.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:03:56.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:56.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:57.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:58.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:58.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:58.313 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:03:58.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:03:58.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:58.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:58.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:59.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:59.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:59.792 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:03:59.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:03:59.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:59.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:04:00.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:01.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:01.260 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:04:01.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:04:01.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:01.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:04:01.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:02.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:02.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:02.734 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:04:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:04:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:02.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:04:04.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:05.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:05.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:05.163 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:04:05.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:04:05.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:05.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:04:05.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:06.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:06.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:06.951 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:04:06.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:04:06.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:06.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:04:07.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:08.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:08.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:08.914 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:04:08.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:04:08.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:08.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:04:09.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:10.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:10.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:10.805 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:04:10.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:04:10.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:10.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:04:11.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:12.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:12.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:12.513 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:04:12.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:04:12.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:12.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:04:13.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:14.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:14.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:14.475 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T09:04:14.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:04:14.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:14.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:04:14.966 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1589027745,v1:127.0.0.1:6827/1589027745] [v2:127.0.0.1:6828/1589027745,v1:127.0.0.1:6829/1589027745] exists,up 293838be-5a59-490f-9336-3c6bca7896d9 2024-04-25T09:04:14.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:04:14.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:04:14.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:04:14.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:04:14.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:04:14.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:04:14.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:04:14.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:04:14.968 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:04:14.968 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:04:14.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:04:14.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:04:14.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:04:15.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:04:15.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:04:15.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:04:15.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:04:15.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:04:15.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:04:15.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:04:15.699 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:15.699 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:04:15.699 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:04:15.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:04:15.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:15.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:04:15.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803810 2024-04-25T09:04:15.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803810 2024-04-25T09:04:15.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810' 2024-04-25T09:04:15.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:15.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:04:16.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607574 2024-04-25T09:04:16.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607574 2024-04-25T09:04:16.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810 1-51539607574' 2024-04-25T09:04:16.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:16.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:04:16.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345934 2024-04-25T09:04:16.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345934 2024-04-25T09:04:16.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810 1-51539607574 2-85899345934' 2024-04-25T09:04:16.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:16.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:04:16.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018894 2024-04-25T09:04:16.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018894 2024-04-25T09:04:16.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810 1-51539607574 2-85899345934 3-128849018894' 2024-04-25T09:04:16.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:16.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803810 2024-04-25T09:04:16.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:16.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:04:16.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803810 2024-04-25T09:04:16.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:16.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803810 2024-04-25T09:04:16.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803810' 2024-04-25T09:04:16.727 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803810 2024-04-25T09:04:16.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:04:17.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803810 -lt 25769803810 2024-04-25T09:04:17.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:17.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607574 2024-04-25T09:04:17.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:17.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:04:17.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607574 2024-04-25T09:04:17.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:17.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607574 2024-04-25T09:04:17.204 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 51539607574 2024-04-25T09:04:17.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607574' 2024-04-25T09:04:17.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:04:17.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607572 -lt 51539607574 2024-04-25T09:04:17.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:04:18.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:04:18.643 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:04:19.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607575 -lt 51539607574 2024-04-25T09:04:19.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:19.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:19.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345934 2024-04-25T09:04:19.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:04:19.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345934 2024-04-25T09:04:19.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:19.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345934 2024-04-25T09:04:19.056 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345934 2024-04-25T09:04:19.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345934' 2024-04-25T09:04:19.057 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:04:19.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345934 -lt 85899345934 2024-04-25T09:04:19.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:19.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018894 2024-04-25T09:04:19.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:04:19.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018894 2024-04-25T09:04:19.475 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:19.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018894 2024-04-25T09:04:19.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018894' 2024-04-25T09:04:19.477 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018894 2024-04-25T09:04:19.477 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:04:19.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018894 -lt 128849018894 2024-04-25T09:04:19.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:04:19.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:19.944 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:20.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:04:20.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:04:20.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:04:20.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:04:20.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:04:20.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:04:20.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:04:20.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:04:20.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:04:20.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:04:20.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:20.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:21.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:04:21.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:04:21.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:04:21.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:172: TEST_rados_get_with_eio: local poolname=pool-rep 2024-04-25T09:04:21.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:173: TEST_rados_get_with_eio: create_pool pool-rep 1 1 2024-04-25T09:04:21.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-04-25T09:04:22.522 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'pool-rep' already exists 2024-04-25T09:04:22.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:04:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:174: TEST_rados_get_with_eio: wait_for_clean 2024-04-25T09:04:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:04:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:04:23.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:04:23.548 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:04:23.548 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:04:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:04:23.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:04:23.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:04:23.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:04:23.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:04:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:04:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:04:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:04:23.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:04:23.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:04:24.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:04:24.085 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:24.085 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:04:24.085 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:04:24.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:04:24.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:24.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:04:24.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803814 2024-04-25T09:04:24.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803814 2024-04-25T09:04:24.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803814' 2024-04-25T09:04:24.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:24.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:04:24.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607578 2024-04-25T09:04:24.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607578 2024-04-25T09:04:24.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803814 1-51539607578' 2024-04-25T09:04:24.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:24.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:04:24.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345938 2024-04-25T09:04:24.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345938 2024-04-25T09:04:24.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803814 1-51539607578 2-85899345938' 2024-04-25T09:04:24.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:24.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:04:24.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018898 2024-04-25T09:04:24.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018898 2024-04-25T09:04:24.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803814 1-51539607578 2-85899345938 3-128849018898' 2024-04-25T09:04:24.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:24.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803814 2024-04-25T09:04:24.713 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:24.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:04:24.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803814 2024-04-25T09:04:24.715 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:24.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803814 2024-04-25T09:04:24.716 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803814 2024-04-25T09:04:24.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803814' 2024-04-25T09:04:24.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:04:25.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803812 -lt 25769803814 2024-04-25T09:04:25.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:04:26.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:04:26.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:04:26.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803814 -lt 25769803814 2024-04-25T09:04:26.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:26.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607578 2024-04-25T09:04:26.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:26.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:04:26.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607578 2024-04-25T09:04:26.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:26.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607578 2024-04-25T09:04:26.529 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 51539607578 2024-04-25T09:04:26.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607578' 2024-04-25T09:04:26.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:04:26.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607578 -lt 51539607578 2024-04-25T09:04:26.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:26.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345938 2024-04-25T09:04:26.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:26.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:04:26.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345938 2024-04-25T09:04:26.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:26.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345938 2024-04-25T09:04:26.951 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345938 2024-04-25T09:04:26.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345938' 2024-04-25T09:04:26.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:04:27.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345938 -lt 85899345938 2024-04-25T09:04:27.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:27.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018898 2024-04-25T09:04:27.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:27.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:04:27.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018898 2024-04-25T09:04:27.377 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:27.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018898 2024-04-25T09:04:27.378 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018898 2024-04-25T09:04:27.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018898' 2024-04-25T09:04:27.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:04:27.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018898 -lt 128849018898 2024-04-25T09:04:27.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:04:27.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:27.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:28.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:04:28.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:04:28.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:04:28.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:04:28.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:04:28.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:04:28.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:04:28.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:04:28.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:04:28.798 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:04:28.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:28.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:04:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:04:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:04:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:175: TEST_rados_get_with_eio: rados_get_data eio td/osd-rep-recov-eio.sh 2024-04-25T09:04:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:111: rados_get_data: local inject=eio 2024-04-25T09:04:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:112: rados_get_data: shift 2024-04-25T09:04:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:113: rados_get_data: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:29.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:115: rados_get_data: local poolname=pool-rep 2024-04-25T09:04:29.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:116: rados_get_data: local objname=obj-eio-388535 2024-04-25T09:04:29.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: get_pg pool-rep obj-eio-388535 2024-04-25T09:04:29.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-04-25T09:04:29.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-eio-388535 2024-04-25T09:04:29.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-eio-388535 2024-04-25T09:04:29.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T09:04:29.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: local pgid=2.0 2024-04-25T09:04:29.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:119: rados_get_data: rados_put td/osd-rep-recov-eio.sh pool-rep obj-eio-388535 2024-04-25T09:04:29.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:29.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:04:29.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-eio-388535 2024-04-25T09:04:29.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:04:29.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:04:29.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:04:29.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:04:29.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:04:29.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:04:29.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:04:29.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:04:29.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-eio-388535 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:04:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:120: rados_get_data: inject_eio rep data pool-rep obj-eio-388535 td/osd-rep-recov-eio.sh 0 2024-04-25T09:04:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:04:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:04:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:04:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:04:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:04:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:04:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-388535 2024-04-25T09:04:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:04:29.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:29.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:04:29.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:04:29.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:04:29.987 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-388535 2024-04-25T09:04:29.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:04:29.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-388535 2024-04-25T09:04:29.988 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-388535 2024-04-25T09:04:29.988 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:04:30.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:04:30.416 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:30.416 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:04:30.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:04:30.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:04:30.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:04:30.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:04:30.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:04:30.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:04:30.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:04:30.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:04:30.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:04:30.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:04:30.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:04:30.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:04:30.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:04:30.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:04:30.419 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:04:30.419 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:04:30.419 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:04:30.419 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:30.419 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:30.419 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:30.419 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:04:30.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:04:30.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:04:30.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:04:30.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:04:30.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:04:30.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:04:30.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:04:30.529 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:30.529 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:30.529 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:30.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:04:30.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:04:30.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-eio-388535 2024-04-25T09:04:30.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:121: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-388535 2024-04-25T09:04:30.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:30.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:04:30.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-388535 2024-04-25T09:04:30.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:04:30.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:04:30.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:04:30.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-388535 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:04:30.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:04:30.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:04:30.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:123: rados_get_data: wait_for_clean 2024-04-25T09:04:30.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:04:30.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:04:30.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:04:30.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:04:30.755 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:04:30.755 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:04:30.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:04:30.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:04:30.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:04:30.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:04:30.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:04:30.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:04:30.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:04:30.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:04:30.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:04:31.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:04:31.307 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:31.308 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:04:31.308 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:04:31.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:04:31.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:31.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:04:31.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803817 2024-04-25T09:04:31.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803817 2024-04-25T09:04:31.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817' 2024-04-25T09:04:31.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:31.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:04:31.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607581 2024-04-25T09:04:31.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607581 2024-04-25T09:04:31.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-51539607581' 2024-04-25T09:04:31.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:31.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:04:31.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345941 2024-04-25T09:04:31.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345941 2024-04-25T09:04:31.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-51539607581 2-85899345941' 2024-04-25T09:04:31.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:31.801 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:04:31.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018901 2024-04-25T09:04:31.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018901 2024-04-25T09:04:31.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-51539607581 2-85899345941 3-128849018901' 2024-04-25T09:04:31.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:31.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803817 2024-04-25T09:04:31.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:31.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:04:31.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803817 2024-04-25T09:04:31.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:31.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803817 2024-04-25T09:04:31.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2024-04-25T09:04:31.951 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803817 2024-04-25T09:04:31.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:04:32.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803817 -lt 25769803817 2024-04-25T09:04:32.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:32.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607581 2024-04-25T09:04:32.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:32.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:04:32.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607581 2024-04-25T09:04:32.358 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:32.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607581 2024-04-25T09:04:32.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607581' 2024-04-25T09:04:32.359 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 51539607581 2024-04-25T09:04:32.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:04:32.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607581 -lt 51539607581 2024-04-25T09:04:32.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:32.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345941 2024-04-25T09:04:32.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:32.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:04:32.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345941 2024-04-25T09:04:32.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:32.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345941 2024-04-25T09:04:32.828 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345941 2024-04-25T09:04:32.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345941' 2024-04-25T09:04:32.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:04:33.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345941 -lt 85899345941 2024-04-25T09:04:33.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:33.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018901 2024-04-25T09:04:33.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:33.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:04:33.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018901 2024-04-25T09:04:33.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:33.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018901 2024-04-25T09:04:33.308 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018901 2024-04-25T09:04:33.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018901' 2024-04-25T09:04:33.308 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:04:33.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018901 -lt 128849018901 2024-04-25T09:04:33.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:04:33.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:33.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:34.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:04:34.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:04:34.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:04:34.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:04:34.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:04:34.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:04:34.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:04:34.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:04:34.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:04:34.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:04:34.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:34.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:35.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:04:35.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:04:35.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:04:35.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: ceph pg 2.0 query 2024-04-25T09:04:35.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:04:35.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: COUNT=1 2024-04-25T09:04:35.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:125: rados_get_data: test 1 = 1 2024-04-25T09:04:35.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:126: rados_get_data: flush_pg_stats 2024-04-25T09:04:35.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:04:35.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:04:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:04:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:04:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:04:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:04:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:35.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:04:35.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803820 2024-04-25T09:04:35.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803820 2024-04-25T09:04:35.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820' 2024-04-25T09:04:35.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:35.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:04:36.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607584 2024-04-25T09:04:36.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607584 2024-04-25T09:04:36.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-51539607584' 2024-04-25T09:04:36.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:36.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:04:36.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345944 2024-04-25T09:04:36.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345944 2024-04-25T09:04:36.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-51539607584 2-85899345944' 2024-04-25T09:04:36.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:36.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:04:36.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018904 2024-04-25T09:04:36.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018904 2024-04-25T09:04:36.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-51539607584 2-85899345944 3-128849018904' 2024-04-25T09:04:36.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:36.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803820 2024-04-25T09:04:36.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:36.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:04:36.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803820 2024-04-25T09:04:36.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:36.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803820 2024-04-25T09:04:36.426 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803820 2024-04-25T09:04:36.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803820' 2024-04-25T09:04:36.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:04:36.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803820 -lt 25769803820 2024-04-25T09:04:36.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:36.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607584 2024-04-25T09:04:36.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:36.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:04:36.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607584 2024-04-25T09:04:36.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:36.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607584 2024-04-25T09:04:36.827 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 51539607584 2024-04-25T09:04:36.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607584' 2024-04-25T09:04:36.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:04:37.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607584 -lt 51539607584 2024-04-25T09:04:37.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:37.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345944 2024-04-25T09:04:37.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:37.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:04:37.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345944 2024-04-25T09:04:37.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:37.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345944 2024-04-25T09:04:37.237 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345944 2024-04-25T09:04:37.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345944' 2024-04-25T09:04:37.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:04:37.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345942 -lt 85899345944 2024-04-25T09:04:37.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:04:38.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:04:38.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:04:39.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345944 -lt 85899345944 2024-04-25T09:04:39.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:39.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:39.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018904 2024-04-25T09:04:39.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:04:39.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018904 2024-04-25T09:04:39.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:39.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018904 2024-04-25T09:04:39.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018904' 2024-04-25T09:04:39.113 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018904 2024-04-25T09:04:39.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:04:39.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018904 -lt 128849018904 2024-04-25T09:04:39.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T09:04:39.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:04:39.924 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-04-25T09:04:39.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: COUNT=1 2024-04-25T09:04:39.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:128: rados_get_data: test 1 = 1 2024-04-25T09:04:39.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: get_osds pool-rep obj-eio-388535 2024-04-25T09:04:39.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:04:39.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-388535 2024-04-25T09:04:39.943 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-388535 2024-04-25T09:04:39.943 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:04:40.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:04:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:40.347 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: object_osds=('3' '1' '0') 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: local object_osds 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:131: rados_get_data: local primary=3 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:132: rados_get_data: local bad_peer=1 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:133: rados_get_data: inject_eio rep data pool-rep obj-eio-388535 td/osd-rep-recov-eio.sh 0 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-388535 2024-04-25T09:04:40.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:04:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:04:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:04:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:04:40.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-388535 2024-04-25T09:04:40.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:04:40.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-388535 2024-04-25T09:04:40.349 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-388535 2024-04-25T09:04:40.350 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:04:40.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:04:40.800 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:40.800 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:04:40.800 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:04:40.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:04:40.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:04:40.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:04:40.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:04:40.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:04:40.802 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:04:40.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:04:40.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:04:40.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:04:40.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:04:40.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:04:40.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:04:40.803 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:04:40.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:04:40.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:04:40.803 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:04:40.803 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:40.804 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:40.804 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:40.804 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:04:40.804 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:04:40.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:04:40.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:04:40.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:04:40.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:04:40.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:04:40.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:04:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:40.921 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:04:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:04:40.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-eio-388535 2024-04-25T09:04:41.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:134: rados_get_data: inject_eio rep data pool-rep obj-eio-388535 td/osd-rep-recov-eio.sh 1 2024-04-25T09:04:41.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:04:41.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:04:41.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:04:41.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:04:41.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:04:41.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:04:41.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-388535 2024-04-25T09:04:41.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:04:41.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:41.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:04:41.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:04:41.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:04:41.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-388535 2024-04-25T09:04:41.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:04:41.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-388535 2024-04-25T09:04:41.029 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-388535 2024-04-25T09:04:41.029 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:04:41.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:04:41.450 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:41.450 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:04:41.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:04:41.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:04:41.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:04:41.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:04:41.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:04:41.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:04:41.452 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:04:41.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:04:41.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:04:41.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:04:41.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:04:41.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:04:41.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:04:41.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:04:41.453 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:04:41.453 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:04:41.453 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:04:41.454 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:41.454 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:41.454 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:41.454 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:04:41.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:04:41.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:04:41.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:04:41.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:04:41.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:04:41.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:04:41.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:04:41.578 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:41.578 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:41.578 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:41.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:04:41.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:04:41.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-eio-388535 2024-04-25T09:04:41.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:136: rados_get_data: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:04:41.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:04:41.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:04:41.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:04:41.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:04:41.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:04:41.690 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:04:41.690 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:04:41.690 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:04:41.691 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:41.691 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:41.691 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:41.691 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:04:41.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:04:41.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:04:41.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:137: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-388535 2024-04-25T09:04:41.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:41.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:04:41.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-388535 2024-04-25T09:04:41.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:04:41.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:04:41.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:04:41.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-388535 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:04:41.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:04:41.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:04:41.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:140: rados_get_data: wait_for_clean 2024-04-25T09:04:41.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:04:41.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:04:41.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:04:41.836 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:04:41.837 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:04:41.837 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:04:41.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:04:41.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:04:41.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:04:41.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:04:41.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:04:41.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:04:41.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:04:41.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:04:41.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:04:42.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:04:42.344 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:42.344 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:04:42.344 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:04:42.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:04:42.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:42.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:04:42.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803823 2024-04-25T09:04:42.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803823 2024-04-25T09:04:42.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823' 2024-04-25T09:04:42.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:42.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:04:42.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607588 2024-04-25T09:04:42.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607588 2024-04-25T09:04:42.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-51539607588' 2024-04-25T09:04:42.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:42.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:04:42.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345947 2024-04-25T09:04:42.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345947 2024-04-25T09:04:42.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-51539607588 2-85899345947' 2024-04-25T09:04:42.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:42.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:04:43.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018907 2024-04-25T09:04:43.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018907 2024-04-25T09:04:43.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-51539607588 2-85899345947 3-128849018907' 2024-04-25T09:04:43.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:43.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803823 2024-04-25T09:04:43.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:43.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:04:43.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803823 2024-04-25T09:04:43.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:43.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803823 2024-04-25T09:04:43.024 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803823 2024-04-25T09:04:43.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803823' 2024-04-25T09:04:43.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:04:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803821 -lt 25769803823 2024-04-25T09:04:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:04:44.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:04:44.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:04:44.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803824 -lt 25769803823 2024-04-25T09:04:44.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:44.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607588 2024-04-25T09:04:44.821 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:44.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:04:44.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:44.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607588 2024-04-25T09:04:44.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607588 2024-04-25T09:04:44.825 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 51539607588 2024-04-25T09:04:44.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607588' 2024-04-25T09:04:44.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:04:45.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607588 -lt 51539607588 2024-04-25T09:04:45.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:45.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:45.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345947 2024-04-25T09:04:45.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:04:45.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345947 2024-04-25T09:04:45.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:45.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345947 2024-04-25T09:04:45.251 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345947 2024-04-25T09:04:45.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345947' 2024-04-25T09:04:45.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:04:45.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345948 -lt 85899345947 2024-04-25T09:04:45.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:45.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018907 2024-04-25T09:04:45.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:45.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:04:45.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018907 2024-04-25T09:04:45.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:45.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018907 2024-04-25T09:04:45.668 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018907 2024-04-25T09:04:45.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018907' 2024-04-25T09:04:45.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:04:46.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018907 -lt 128849018907 2024-04-25T09:04:46.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:04:46.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:46.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:46.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:04:46.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:04:46.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:04:46.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:04:46.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:04:46.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:04:46.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:04:46.686 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:04:47.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:04:47.154 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:04:47.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:47.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:47.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:04:47.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:04:47.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:04:47.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:142: rados_get_data: inject_eio rep data pool-rep obj-eio-388535 td/osd-rep-recov-eio.sh 0 2024-04-25T09:04:47.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:04:47.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-388535 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-388535 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:04:47.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-388535 2024-04-25T09:04:47.670 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-388535 2024-04-25T09:04:47.670 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:04:48.147 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:04:48.148 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:48.148 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:04:48.148 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:04:48.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:04:48.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:04:48.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:04:48.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:04:48.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:04:48.149 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:04:48.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:04:48.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:04:48.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:04:48.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:04:48.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:04:48.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:04:48.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:04:48.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:04:48.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:04:48.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:04:48.151 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:48.151 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:48.151 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:48.151 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:04:48.151 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:04:48.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:04:48.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:04:48.262 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:04:48.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:04:48.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:04:48.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:04:48.263 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:48.263 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:48.263 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:48.263 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:04:48.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:04:48.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-eio-388535 2024-04-25T09:04:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:143: rados_get_data: inject_eio rep data pool-rep obj-eio-388535 td/osd-rep-recov-eio.sh 2 2024-04-25T09:04:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:04:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:04:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:04:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:04:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:04:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:04:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-388535 2024-04-25T09:04:48.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:04:48.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:48.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:04:48.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=2 2024-04-25T09:04:48.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:04:48.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-388535 2024-04-25T09:04:48.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:04:48.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-388535 2024-04-25T09:04:48.374 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-388535 2024-04-25T09:04:48.374 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:04:48.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:04:48.791 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:48.791 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:04:48.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:04:48.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:04:48.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:04:48.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-04-25T09:04:48.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:04:48.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:04:48.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-04-25T09:04:48.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:04:48.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-25T09:04:48.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:04:48.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-25T09:04:48.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:04:48.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:04:48.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:04:48.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-25T09:04:48.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T09:04:48.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T09:04:48.794 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:48.794 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:48.794 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:48.794 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.0.asok 2024-04-25T09:04:48.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:04:48.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:04:48.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:04:48.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:04:48.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-04-25T09:04:48.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T09:04:48.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T09:04:48.910 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:48.910 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:48.910 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:48.910 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.0.asok 2024-04-25T09:04:48.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:04:48.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.0.asok injectdataerr pool-rep obj-eio-388535 2024-04-25T09:04:49.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:144: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-388535 2024-04-25T09:04:49.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:49.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:04:49.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-388535 2024-04-25T09:04:49.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:04:49.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:04:49.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:04:49.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-388535 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:04:49.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:04:49.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:04:49.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:146: rados_get_data: wait_for_clean 2024-04-25T09:04:49.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:04:49.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:04:49.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:04:49.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:04:49.066 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:04:49.066 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:04:49.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:04:49.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:04:49.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:04:49.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:04:49.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:04:49.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:04:49.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:04:49.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:04:49.140 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:04:49.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:04:49.517 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:49.517 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:04:49.517 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:04:49.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:04:49.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:49.518 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:04:49.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803827 2024-04-25T09:04:49.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803827 2024-04-25T09:04:49.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803827' 2024-04-25T09:04:49.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:49.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:04:49.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607591 2024-04-25T09:04:49.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607591 2024-04-25T09:04:49.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803827 1-51539607591' 2024-04-25T09:04:49.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:49.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:04:50.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345951 2024-04-25T09:04:50.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345951 2024-04-25T09:04:50.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803827 1-51539607591 2-85899345951' 2024-04-25T09:04:50.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:50.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:04:50.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018911 2024-04-25T09:04:50.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018911 2024-04-25T09:04:50.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803827 1-51539607591 2-85899345951 3-128849018911' 2024-04-25T09:04:50.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:50.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803827 2024-04-25T09:04:50.169 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:50.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:04:50.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803827 2024-04-25T09:04:50.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:50.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803827 2024-04-25T09:04:50.173 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803827 2024-04-25T09:04:50.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-04-25T09:04:50.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:04:50.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803827 -lt 25769803827 2024-04-25T09:04:50.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:50.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:50.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607591 2024-04-25T09:04:50.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:04:50.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607591 2024-04-25T09:04:50.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:50.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607591 2024-04-25T09:04:50.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607591' 2024-04-25T09:04:50.561 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 51539607591 2024-04-25T09:04:50.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:04:51.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607591 -lt 51539607591 2024-04-25T09:04:51.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:51.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345951 2024-04-25T09:04:51.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:51.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:04:51.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345951 2024-04-25T09:04:51.009 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:51.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345951 2024-04-25T09:04:51.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345951' 2024-04-25T09:04:51.011 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345951 2024-04-25T09:04:51.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:04:51.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345951 -lt 85899345951 2024-04-25T09:04:51.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:51.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018911 2024-04-25T09:04:51.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:51.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:04:51.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018911 2024-04-25T09:04:51.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:51.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018911 2024-04-25T09:04:51.417 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018911 2024-04-25T09:04:51.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018911' 2024-04-25T09:04:51.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:04:51.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018910 -lt 128849018911 2024-04-25T09:04:51.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:04:52.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:04:52.838 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:04:53.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018911 -lt 128849018911 2024-04-25T09:04:53.237 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:04:53.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:53.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:53.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:04:53.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:04:53.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:04:53.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:04:53.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:04:53.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:04:53.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:04:53.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:04:54.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:04:54.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:04:54.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:04:54.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:04:54.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:04:54.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:04:54.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:04:54.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: ceph pg 2.0 query 2024-04-25T09:04:54.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:04:54.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: COUNT=3 2024-04-25T09:04:54.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:148: rados_get_data: test 3 = 3 2024-04-25T09:04:54.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:149: rados_get_data: flush_pg_stats 2024-04-25T09:04:54.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:04:54.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:04:55.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:04:55.276 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:55.276 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:04:55.276 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:04:55.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:04:55.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:55.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:04:55.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803830 2024-04-25T09:04:55.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803830 2024-04-25T09:04:55.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830' 2024-04-25T09:04:55.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:55.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:04:55.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607594 2024-04-25T09:04:55.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607594 2024-04-25T09:04:55.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830 1-51539607594' 2024-04-25T09:04:55.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:55.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:04:55.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345954 2024-04-25T09:04:55.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345954 2024-04-25T09:04:55.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830 1-51539607594 2-85899345954' 2024-04-25T09:04:55.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:04:55.722 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:04:55.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018914 2024-04-25T09:04:55.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018914 2024-04-25T09:04:55.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830 1-51539607594 2-85899345954 3-128849018914' 2024-04-25T09:04:55.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:55.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:55.858 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803830 2024-04-25T09:04:55.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:04:55.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803830 2024-04-25T09:04:55.859 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:55.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803830 2024-04-25T09:04:55.860 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803830 2024-04-25T09:04:55.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-04-25T09:04:55.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:04:56.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803828 -lt 25769803830 2024-04-25T09:04:56.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:04:57.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:04:57.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:04:57.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803830 -lt 25769803830 2024-04-25T09:04:57.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:57.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607594 2024-04-25T09:04:57.646 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:57.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:04:57.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607594 2024-04-25T09:04:57.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:57.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607594 2024-04-25T09:04:57.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607594' 2024-04-25T09:04:57.651 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 51539607594 2024-04-25T09:04:57.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:04:58.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607594 -lt 51539607594 2024-04-25T09:04:58.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:58.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345954 2024-04-25T09:04:58.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:58.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:04:58.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345954 2024-04-25T09:04:58.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:58.115 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345954 2024-04-25T09:04:58.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345954 2024-04-25T09:04:58.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345954' 2024-04-25T09:04:58.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:04:58.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345954 -lt 85899345954 2024-04-25T09:04:58.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:04:58.596 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018914 2024-04-25T09:04:58.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:04:58.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:04:58.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018914 2024-04-25T09:04:58.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:04:58.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018914 2024-04-25T09:04:58.601 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018914 2024-04-25T09:04:58.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018914' 2024-04-25T09:04:58.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:04:59.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018914 -lt 128849018914 2024-04-25T09:04:59.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T09:04:59.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:04:59.479 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-04-25T09:04:59.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: COUNT=4 2024-04-25T09:04:59.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:151: rados_get_data: test 4 = 4 2024-04-25T09:04:59.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:153: rados_get_data: inject_eio rep data pool-rep obj-eio-388535 td/osd-rep-recov-eio.sh 0 2024-04-25T09:04:59.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:04:59.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:04:59.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:04:59.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:04:59.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:04:59.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:04:59.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-388535 2024-04-25T09:04:59.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:04:59.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:04:59.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:04:59.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:04:59.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:04:59.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-388535 2024-04-25T09:04:59.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:04:59.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-388535 2024-04-25T09:04:59.498 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-388535 2024-04-25T09:04:59.498 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:04:59.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:04:59.895 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:04:59.895 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:04:59.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:04:59.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:04:59.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:04:59.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:04:59.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:04:59.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:04:59.896 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:04:59.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:04:59.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:04:59.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:04:59.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:04:59.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:04:59.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:04:59.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:04:59.899 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:04:59.899 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:04:59.899 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:04:59.899 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:04:59.899 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:59.899 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:04:59.899 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:04:59.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:05:00.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:05:00.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:05:00.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:05:00.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:05:00.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:05:00.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:05:00.011 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:05:00.011 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:05:00.011 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:05:00.011 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:05:00.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:05:00.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-eio-388535 2024-04-25T09:05:00.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:154: rados_get_data: inject_eio rep data pool-rep obj-eio-388535 td/osd-rep-recov-eio.sh 1 2024-04-25T09:05:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:05:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:05:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:05:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:05:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:05:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:05:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-388535 2024-04-25T09:05:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:05:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:05:00.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:05:00.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:05:00.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:05:00.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-388535 2024-04-25T09:05:00.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:05:00.120 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-388535 2024-04-25T09:05:00.120 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-388535 2024-04-25T09:05:00.121 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:05:00.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:05:00.540 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:05:00.541 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:05:00.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:05:00.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:05:00.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:05:00.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:05:00.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:05:00.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:05:00.541 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:05:00.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:05:00.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:05:00.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:05:00.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:05:00.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:05:00.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:05:00.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:05:00.543 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:05:00.543 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:05:00.543 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:05:00.544 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:05:00.544 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:05:00.544 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:05:00.544 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:05:00.545 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:05:00.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:05:00.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:05:00.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:05:00.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:05:00.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:05:00.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:05:00.664 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:05:00.664 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:05:00.664 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:05:00.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:05:00.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:05:00.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-eio-388535 2024-04-25T09:05:00.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:155: rados_get_data: inject_eio rep data pool-rep obj-eio-388535 td/osd-rep-recov-eio.sh 2 2024-04-25T09:05:00.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:05:00.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:05:00.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-388535 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=2 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-388535 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:05:00.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-388535 2024-04-25T09:05:00.784 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-388535 2024-04-25T09:05:00.784 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:05:01.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:05:01.199 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:05:01.199 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:05:01.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:05:01.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:05:01.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:05:01.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-04-25T09:05:01.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:05:01.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:05:01.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-04-25T09:05:01.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:05:01.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-25T09:05:01.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:05:01.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-25T09:05:01.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:05:01.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:05:01.202 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:05:01.203 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-25T09:05:01.203 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T09:05:01.203 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T09:05:01.203 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:05:01.203 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:05:01.203 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:05:01.203 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.0.asok 2024-04-25T09:05:01.203 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:05:01.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:05:01.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:05:01.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:05:01.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-04-25T09:05:01.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T09:05:01.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T09:05:01.328 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:05:01.328 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:05:01.328 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:05:01.328 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.0.asok 2024-04-25T09:05:01.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:05:01.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.0.asok injectdataerr pool-rep obj-eio-388535 2024-04-25T09:05:01.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:156: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-388535 hang 2024-04-25T09:05:01.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:05:01.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:05:01.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-388535 2024-04-25T09:05:01.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=hang 2024-04-25T09:05:01.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' hang = fail ']' 2024-04-25T09:05:01.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' hang = hang ']' 2024-04-25T09:05:01.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:98: rados_get: timeout 5 rados --pool pool-rep get obj-eio-388535 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:05:06.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:99: rados_get: test 124 = 124 2024-04-25T09:05:06.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:100: rados_get: return 2024-04-25T09:05:06.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:158: rados_get_data: wait_for_clean 2024-04-25T09:05:06.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:05:06.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:05:06.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:05:06.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:05:06.451 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:05:06.451 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:05:06.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:05:06.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:05:06.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:05:06.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:05:06.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:05:06.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:05:06.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:05:06.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:05:06.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:05:06.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:05:06.992 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:05:06.992 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:05:06.992 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:05:06.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:05:06.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:06.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:05:07.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803834 2024-04-25T09:05:07.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803834 2024-04-25T09:05:07.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803834' 2024-04-25T09:05:07.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:07.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:05:07.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607598 2024-04-25T09:05:07.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607598 2024-04-25T09:05:07.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803834 1-51539607598' 2024-04-25T09:05:07.312 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:07.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:05:07.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345958 2024-04-25T09:05:07.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345958 2024-04-25T09:05:07.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803834 1-51539607598 2-85899345958' 2024-04-25T09:05:07.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:07.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:05:07.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018918 2024-04-25T09:05:07.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018918 2024-04-25T09:05:07.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803834 1-51539607598 2-85899345958 3-128849018918' 2024-04-25T09:05:07.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:07.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803834 2024-04-25T09:05:07.608 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:07.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:05:07.609 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803834 2024-04-25T09:05:07.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:07.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803834 2024-04-25T09:05:07.611 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803834 2024-04-25T09:05:07.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803834' 2024-04-25T09:05:07.611 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:05:08.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803832 -lt 25769803834 2024-04-25T09:05:08.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:05:09.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:05:09.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:05:09.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803835 -lt 25769803834 2024-04-25T09:05:09.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:09.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607598 2024-04-25T09:05:09.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:09.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:05:09.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607598 2024-04-25T09:05:09.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:09.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607598 2024-04-25T09:05:09.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607598' 2024-04-25T09:05:09.432 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 51539607598 2024-04-25T09:05:09.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:05:09.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607599 -lt 51539607598 2024-04-25T09:05:09.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:09.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345958 2024-04-25T09:05:09.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:09.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:05:09.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345958 2024-04-25T09:05:09.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:09.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345958 2024-04-25T09:05:09.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345958' 2024-04-25T09:05:09.896 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345958 2024-04-25T09:05:09.897 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:05:10.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345959 -lt 85899345958 2024-04-25T09:05:10.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:10.330 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018918 2024-04-25T09:05:10.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:10.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:05:10.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018918 2024-04-25T09:05:10.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:10.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018918 2024-04-25T09:05:10.332 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018918 2024-04-25T09:05:10.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018918' 2024-04-25T09:05:10.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:05:10.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018919 -lt 128849018918 2024-04-25T09:05:10.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:05:10.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:10.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:11.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:05:11.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:11.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:11.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:11.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:11.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:11.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:11.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:11.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:05:11.727 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:11.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:11.728 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:05:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-04-25T09:05:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:05:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-04-25T09:05:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:05:12.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:05:12.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:05:12.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:12.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:12.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:12.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:12.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:12.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:12.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:12.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:05:12.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:12.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:12.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:13.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:05:13.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:05:13.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:05:13.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:05:13.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:05:13.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:05:13.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:05:13.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:05:13.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:05:13.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:05:13.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:05:13.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T09:05:13.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:05:13.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T09:05:14.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:05:14.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:14.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:14.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:14.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:14.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:14.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:14.073 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:14.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:05:14.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:14.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:14.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:15.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:05:15.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:05:15.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:05:15.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:05:15.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:05:15.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:05:15.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:05:15.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:05:15.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:05:15.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:05:15.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:05:15.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-25T09:05:15.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:05:15.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T09:05:15.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:05:15.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:15.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:15.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:15.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:15.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:15.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:15.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:16.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:05:16.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:16.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:16.361 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:16.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:05:16.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:05:16.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:05:16.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:05:16.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:05:16.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:05:16.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:05:16.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:05:16.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:05:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:05:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:05:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-04-25T09:05:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:05:17.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T09:05:18.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:05:18.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:18.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:18.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:18.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:18.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:18.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:18.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:18.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:05:18.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:18.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:18.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:19.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:05:19.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:05:19.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:05:19.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:05:19.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:05:19.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:05:19.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:05:19.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:05:19.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:05:19.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:05:19.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:05:19.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 13 )) 2024-04-25T09:05:19.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:05:19.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T09:05:21.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:05:21.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:21.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:21.296 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:21.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:21.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:21.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:21.297 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:21.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:05:21.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:21.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:21.699 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:22.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:05:22.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:05:22.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:05:22.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:05:22.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:05:22.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:05:22.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:05:22.238 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:05:22.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:05:22.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:05:22.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:05:22.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 13 )) 2024-04-25T09:05:22.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:05:22.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T09:05:25.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:05:25.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:25.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:25.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:25.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:25.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:25.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:25.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:26.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:05:26.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:26.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:26.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:26.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:05:26.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:05:26.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:05:26.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:05:26.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:05:26.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:05:26.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:05:26.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:05:26.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:05:27.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:05:27.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:05:27.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 13 )) 2024-04-25T09:05:27.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:05:27.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T09:05:33.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:05:33.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:33.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:33.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:33.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:33.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:33.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:33.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:34.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:05:34.365 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:34.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:34.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:34.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:05:34.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:05:34.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:05:34.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:05:34.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:05:34.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:05:34.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:05:34.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:05:34.905 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:05:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:05:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:05:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 13 )) 2024-04-25T09:05:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:05:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 12.8 2024-04-25T09:05:48.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:05:48.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:48.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:48.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:48.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:48.264 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:48.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:48.265 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:48.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:05:48.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:48.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:48.683 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:49.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:05:49.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:05:49.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:05:49.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:05:49.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:05:49.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:05:49.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:05:49.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:05:49.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:05:49.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:05:49.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:05:49.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 13 )) 2024-04-25T09:05:49.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:05:49.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T09:06:04.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:04.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:04.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:04.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:04.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:04.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:04.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:04.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:05.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:06:05.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:05.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:05.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:05.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:06:05.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:06:05.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:05.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:05.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:05.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:05.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:05.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:05.684 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:06.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:06:06.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:06:06.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 9 >= 13 )) 2024-04-25T09:06:06.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:06.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T09:06:21.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:21.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:21.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:21.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:21.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:21.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:21.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:21.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:21.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:06:21.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:21.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:21.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:22.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:06:22.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:06:22.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:22.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:22.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:22.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:22.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:22.182 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:22.182 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:22.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:06:22.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:06:22.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 10 >= 13 )) 2024-04-25T09:06:22.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:22.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T09:06:37.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:37.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:37.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:37.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:37.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:37.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:37.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:37.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:38.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:06:38.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:38.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:38.189 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:06:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:06:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:38.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:38.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:38.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:38.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:38.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:38.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:39.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:06:39.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:06:39.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 11 >= 13 )) 2024-04-25T09:06:39.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:39.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T09:06:54.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:54.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:54.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:54.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:54.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:54.272 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:54.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:54.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:54.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:06:54.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:54.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:54.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:55.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:06:55.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:06:55.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:55.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:55.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:55.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:55.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:55.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:55.233 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:55.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:06:55.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:06:55.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 12 >= 13 )) 2024-04-25T09:06:55.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:55.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 4.5 2024-04-25T09:07:00.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:07:00.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:07:00.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:07:00.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:07:00.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:07:00.244 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:07:00.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:07:00.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:07:00.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:07:00.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:07:00.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:07:00.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:07:01.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:07:01.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:07:01.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:07:01.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:07:01.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:07:01.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:07:01.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:07:01.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:07:01.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:07:01.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:07:01.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:07:01.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 13 >= 13 )) 2024-04-25T09:07:01.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T09:07:02.363 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T09:07:02.363 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_fingerprint": "c842dcb0-dadb-4145-8c91-9b162924a248", 2024-04-25T09:07:02.363 INFO:tasks.workunit.client.0.smithi102.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T09:07:02.363 INFO:tasks.workunit.client.0.smithi102.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T09:07:02.363 INFO:tasks.workunit.client.0.smithi102.stdout: "timestamp": "2024-04-25T09:07:02.373257+0000", 2024-04-25T09:07:02.363 INFO:tasks.workunit.client.0.smithi102.stdout: "tag": "", 2024-04-25T09:07:02.363 INFO:tasks.workunit.client.0.smithi102.stdout: "health": { 2024-04-25T09:07:02.363 INFO:tasks.workunit.client.0.smithi102.stdout: "status": "HEALTH_ERR", 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: "checks": { 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: "OBJECT_UNFOUND": { 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1/1 objects unfound (100.000%)", 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 2.0 has 1 unfound objects" 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: "PG_DAMAGED": { 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_ERR", 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "Possible data damage: 1 pg recovery_unfound", 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.365 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: "PG_DEGRADED": { 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "Degraded data redundancy: 3/3 objects degraded (100.000%), 1 pg degraded", 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.366 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "2 pool(s) do not have an application enabled", 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 2 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [ 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "application not enabled on pool 'foo'" 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.367 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "application not enabled on pool 'pool-rep'" 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: "SLOW_OPS": { 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: "summary": { 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: "message": "1 slow ops, oldest one blocked for 117 sec, osd.3 has slow ops", 2024-04-25T09:07:02.368 INFO:tasks.workunit.client.0.smithi102.stdout: "count": 1 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: "detail": [], 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: "muted": false 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: "mutes": [] 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_first_committed": 1, 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap_last_committed": 1, 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: "monmap": { 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "3ad78387-2d96-4ac5-b314-cb95cfa29968", 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T09:01:33.682868Z", 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T09:01:33.682868Z", 2024-04-25T09:07:02.369 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release": 19, 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "min_mon_release_name": "squid", 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "election_strategy": 1, 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "disallowed_leaders: ": "", 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": false, 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "tiebreaker_mon": "", 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_ranks: ": "", 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "features": { 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "persistent": [ 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "kraken", 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "luminous", 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "mimic", 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap-prune", 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "nautilus", 2024-04-25T09:07:02.370 INFO:tasks.workunit.client.0.smithi102.stdout: "octopus", 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "pacific", 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "elector-pinging", 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "quincy", 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "reef", 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "squid" 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "optional": [] 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "mons": [ 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "rank": 0, 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "a", 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.371 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7140", 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 0 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:7140/0", 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:7140/0", 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: "priority": 0, 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 0, 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_location": "{}" 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: "quorum": [ 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: 0 2024-04-25T09:07:02.372 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap": { 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 37, 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "fsid": "3ad78387-2d96-4ac5-b314-cb95cfa29968", 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "created": "2024-04-25T09:01:34.766683+0000", 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T09:04:23.212775+0000", 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "last_up_change": "2024-04-25T09:04:13.901398+0000", 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "last_in_change": "2024-04-25T09:03:34.287827+0000", 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_num": 5799936, 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_set": [ 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "pglog_hardlimit", 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snapdirs", 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "recovery_deletes", 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: "sortbitwise" 2024-04-25T09:07:02.373 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_version": 9, 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_snapshot": "", 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_max": 2, 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "max_osd": 4, 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "require_min_compat_client": "luminous", 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "min_compat_client": "jewel", 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "require_osd_release": "squid", 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "allow_crimson": false, 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "pools": [ 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T09:07:02.374 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_name": "foo", 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "create_time": "2024-04-25T09:01:35.761726+0000", 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": 1, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_names": "hashpspool", 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 3, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "min_size": 2, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_rule": 0, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_count": 0, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_target": 0, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "object_hash": 2, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_autoscale_mode": "off", 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num": 8, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num": 8, 2024-04-25T09:07:02.375 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num_target": 8, 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_target": 8, 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_pending": 8, 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pg_merge_meta": { 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "source_pgid": "0.0", 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "ready_epoch": 0, 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_started": 0, 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": 0, 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "source_version": "0'0", 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "target_version": "0'0" 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "last_change": "8", 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend": "0", 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T09:07:02.376 INFO:tasks.workunit.client.0.smithi102.stdout: "auid": 0, 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_mode": "selfmanaged", 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_seq": 0, 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_epoch": 0, 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_snaps": [], 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps": "[]", 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_bytes": 0, 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_objects": 0, 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "tiers": [], 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "tier_of": -1, 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "read_tier": -1, 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "write_tier": -1, 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_mode": "none", 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_bytes": 0, 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_objects": 0, 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T09:07:02.377 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_flush_age": 0, 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_evict_age": 0, 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profile": "", 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_params": { 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "none" 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_period": 0, 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_count": 0, 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "use_gmt_hitset": true, 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "min_read_recency_for_promote": 0, 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "min_write_recency_for_promote": 0, 2024-04-25T09:07:02.378 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_search_last_n": 0, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "grade_table": [], 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "stripe_width": 0, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_num_objects": 0, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "fast_read": false, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "options": {}, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "application_metadata": {}, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "read_balance": { 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "score_type": "Fair distribution", 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "score_acting": 1.5, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "score_stable": 1.5, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_score": 1, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_acting": 1.5, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_stable": 1.5, 2024-04-25T09:07:02.379 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity_weighted": 1, 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity": 1, 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity_weighted": 1 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 2, 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_name": "pool-rep", 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "create_time": "2024-04-25T09:04:21.849708+0000", 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "flags": 1, 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "flags_names": "hashpspool", 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "size": 3, 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "min_size": 2, 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_rule": 0, 2024-04-25T09:07:02.380 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_count": 0, 2024-04-25T09:07:02.381 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_target": 0, 2024-04-25T09:07:02.381 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T09:07:02.381 INFO:tasks.workunit.client.0.smithi102.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T09:07:02.381 INFO:tasks.workunit.client.0.smithi102.stdout: "object_hash": 2, 2024-04-25T09:07:02.381 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_autoscale_mode": "off", 2024-04-25T09:07:02.381 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num": 1, 2024-04-25T09:07:02.381 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num": 1, 2024-04-25T09:07:02.381 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_placement_num_target": 1, 2024-04-25T09:07:02.381 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_target": 1, 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_num_pending": 1, 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pg_merge_meta": { 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "source_pgid": "0.0", 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "ready_epoch": 0, 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_started": 0, 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": 0, 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "source_version": "0'0", 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "target_version": "0'0" 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "last_change": "37", 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend": "0", 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T09:07:02.382 INFO:tasks.workunit.client.0.smithi102.stdout: "auid": 0, 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_mode": "selfmanaged", 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_seq": 0, 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_epoch": 0, 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_snaps": [], 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps": "[]", 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_bytes": 0, 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "quota_max_objects": 0, 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "tiers": [], 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "tier_of": -1, 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "read_tier": -1, 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "write_tier": -1, 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_mode": "none", 2024-04-25T09:07:02.383 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_bytes": 0, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "target_max_objects": 0, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_flush_age": 0, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "cache_min_evict_age": 0, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profile": "", 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_params": { 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "none" 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_period": 0, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_count": 0, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "use_gmt_hitset": true, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "min_read_recency_for_promote": 0, 2024-04-25T09:07:02.384 INFO:tasks.workunit.client.0.smithi102.stdout: "min_write_recency_for_promote": 0, 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "hit_set_search_last_n": 0, 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "grade_table": [], 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "stripe_width": 0, 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "expected_num_objects": 0, 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "fast_read": false, 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "options": {}, 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "application_metadata": {}, 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "read_balance": { 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "score_type": "Fair distribution", 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "score_acting": 4, 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "score_stable": 4, 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_score": 0.75, 2024-04-25T09:07:02.385 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_acting": 3, 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: "raw_score_stable": 3, 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity_weighted": 1, 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity": 1, 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: "average_primary_affinity_weighted": 1 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: "osds": [ 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "c68c3860-5dc2-4870-ba5d-ee8504bed474", 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T09:07:02.386 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 0, 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 0, 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 6, 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 32, 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 0, 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6802", 2024-04-25T09:07:02.387 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 468211697 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6803", 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 468211697 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.388 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.389 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6804", 2024-04-25T09:07:02.389 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 468211697 2024-04-25T09:07:02.389 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.389 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.389 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.389 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6805", 2024-04-25T09:07:02.389 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 468211697 2024-04-25T09:07:02.389 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.389 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.389 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6808", 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 468211697 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6809", 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 468211697 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.390 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6806", 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 468211697 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6807", 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 468211697 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6803/468211697", 2024-04-25T09:07:02.391 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6805/468211697", 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6809/468211697", 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6807/468211697", 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "a550677e-0511-4344-ac39-d294b38ab227", 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 0, 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 0, 2024-04-25T09:07:02.392 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 12, 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 32, 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 0, 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6810", 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1806503951 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6811", 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1806503951 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.393 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6812", 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1806503951 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6813", 2024-04-25T09:07:02.394 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1806503951 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6816", 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1806503951 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6817", 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1806503951 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.395 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.396 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.396 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T09:07:02.396 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.396 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.396 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.396 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6814", 2024-04-25T09:07:02.396 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1806503951 2024-04-25T09:07:02.396 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.396 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6815", 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1806503951 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6811/1806503951", 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6813/1806503951", 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1806503951", 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1806503951", 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.397 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "7ea6236a-4be7-4a6b-98b9-d8ecb4096f02", 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 0, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 0, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 20, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 32, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 0, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T09:07:02.398 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6818", 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2777005445 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6819", 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2777005445 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.399 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6820", 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2777005445 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6821", 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2777005445 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.400 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6824", 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2777005445 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6825", 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2777005445 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6822", 2024-04-25T09:07:02.401 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2777005445 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6823", 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 2777005445 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6819/2777005445", 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6821/2777005445", 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6825/2777005445", 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6823/2777005445", 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T09:07:02.402 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 3, 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "uuid": "293838be-5a59-490f-9336-3c6bca7896d9", 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 1, 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "in": 1, 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 1, 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_affinity": 1, 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_begin": 0, 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "last_clean_end": 0, 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 30, 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "up_thru": 35, 2024-04-25T09:07:02.403 INFO:tasks.workunit.client.0.smithi102.stdout: "down_at": 0, 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: "lost_at": 0, 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addrs": { 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6826", 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1589027745 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6827", 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1589027745 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.404 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addrs": { 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6828", 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1589027745 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6829", 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1589027745 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addrs": { 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.405 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6832", 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1589027745 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6833", 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1589027745 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addrs": { 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: "addrvec": [ 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.406 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v2", 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6830", 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1589027745 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "v1", 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: "addr": "127.0.0.1:6831", 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: "nonce": 1589027745 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: "public_addr": "127.0.0.1:6827/1589027745", 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: "cluster_addr": "127.0.0.1:6829/1589027745", 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_back_addr": "127.0.0.1:6833/1589027745", 2024-04-25T09:07:02.407 INFO:tasks.workunit.client.0.smithi102.stdout: "heartbeat_front_addr": "127.0.0.1:6831/1589027745", 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "state": [ 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "exists", 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "up" 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_xinfo": [ 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "0.000000", 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T09:02:11.718904+0000", 2024-04-25T09:07:02.408 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "0.000000", 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T09:02:51.274553+0000", 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "0.000000", 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T09:07:02.409 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T09:03:32.070776+0000", 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 3, 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: "down_stamp": "0.000000", 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_probability": 0, 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: "laggy_interval": 0, 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: "features": 4540701547738038271, 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: "old_weight": 0, 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: "last_purged_snaps_scrub": "2024-04-25T09:04:13.480337+0000", 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: "dead_epoch": 0 2024-04-25T09:07:02.410 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap": [], 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_items": [], 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_upmap_primaries": [], 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "pg_temp": [], 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "primary_temp": [], 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "blocklist": {}, 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "range_blocklist": {}, 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "erasure_code_profiles": { 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "default": { 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "k": "2", 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "m": "2", 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "plugin": "jerasure", 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: "technique": "reed_sol_van" 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.411 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "removed_snaps_queue": [], 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "new_removed_snaps": [], 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "new_purged_snaps": [], 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "crush_node_flags": {}, 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "device_class_flags": {}, 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode": { 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_enabled": false, 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_bucket_count": 0, 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "degraded_stretch_mode": 0, 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "recovering_stretch_mode": 0, 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "stretch_mode_bucket": 0 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.412 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_metadata": [ 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6804/468211697,v1:127.0.0.1:6805/468211697]", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:07:02.413 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/0/block", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T09:01:42.092177Z", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T09:07:02.414 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6802/468211697,v1:127.0.0.1:6803/468211697]", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/468211697,v1:127.0.0.1:6809/468211697]", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/468211697,v1:127.0.0.1:6807/468211697]", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:07:02.415 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/osd-rep-recov-eio.sh/0", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6812/1806503951,v1:127.0.0.1:6813/1806503951]", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:07:02.416 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/1/block", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T09:02:20.316749Z", 2024-04-25T09:07:02.417 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6810/1806503951,v1:127.0.0.1:6811/1806503951]", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1806503951,v1:127.0.0.1:6817/1806503951]", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1806503951,v1:127.0.0.1:6815/1806503951]", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:07:02.418 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/osd-rep-recov-eio.sh/1", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6820/2777005445,v1:127.0.0.1:6821/2777005445]", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T09:07:02.419 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/2/block", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:07:02.420 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T09:03:00.118319Z", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6818/2777005445,v1:127.0.0.1:6819/2777005445]", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/2777005445,v1:127.0.0.1:6825/2777005445]", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/2777005445,v1:127.0.0.1:6823/2777005445]", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T09:07:02.421 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/osd-rep-recov-eio.sh/2", 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.422 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 3, 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "arch": "x86_64", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "back_addr": "[v2:127.0.0.1:6828/1589027745,v1:127.0.0.1:6829/1589027745]", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "back_iface": "", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs": "1", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/3/block", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:07:02.423 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_release": "squid", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "created_at": "2024-04-25T09:03:41.243750Z", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "default_device_class": "hdd", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5QPPM", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": "sda", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "distro": "ubuntu", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "distro_version": "22.04", 2024-04-25T09:07:02.424 INFO:tasks.workunit.client.0.smithi102.stdout: "front_addr": "[v2:127.0.0.1:6826/1589027745,v1:127.0.0.1:6827/1589027745]", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "front_iface": "", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_back_addr": "[v2:127.0.0.1:6832/1589027745,v1:127.0.0.1:6833/1589027745]", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_front_addr": "[v2:127.0.0.1:6830/1589027745,v1:127.0.0.1:6831/1589027745]", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "hostname": "smithi102", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "journal_rotational": "1", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "mem_total_kb": "32746856", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "os": "Linux", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_data": "td/osd-rep-recov-eio.sh/3", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "osdspec_affinity": "", 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: "rotational": "1" 2024-04-25T09:07:02.425 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_clean_epochs": { 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: "min_last_epoch_clean": 0, 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: "last_epoch_clean": { 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: "per_pool": [ 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: "floor": 4294967295 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_epochs": [ 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T09:07:02.426 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 7 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 13 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 20 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 3, 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 30 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_first_committed": 1, 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: "osdmap_last_committed": 37, 2024-04-25T09:07:02.427 INFO:tasks.workunit.client.0.smithi102.stdout: "crushmap": { 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: "devices": [ 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.0", 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.1", 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.2", 2024-04-25T09:07:02.428 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 3, 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd.3", 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: "class": "hdd" 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: "types": [ 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 0, 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "osd" 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "host" 2024-04-25T09:07:02.429 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 2, 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "chassis" 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 3, 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "rack" 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 4, 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "row" 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 5, 2024-04-25T09:07:02.430 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pdu" 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 6, 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "pod" 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 7, 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "room" 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 8, 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "datacenter" 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 9, 2024-04-25T09:07:02.431 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "zone" 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 10, 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "region" 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "root" 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: "buckets": [ 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -1, 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default", 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T09:07:02.432 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 25608, 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 25608, 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -2, 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "default~hdd", 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 11, 2024-04-25T09:07:02.433 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "root", 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 25608, 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 25608, 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -3, 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102", 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T09:07:02.434 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 25608, 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.435 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 3, 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 3 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: "id": -4, 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: "name": "smithi102~hdd", 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: "type_id": 1, 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: "type_name": "host", 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 25608, 2024-04-25T09:07:02.436 INFO:tasks.workunit.client.0.smithi102.stdout: "alg": "straw2", 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: "hash": "rjenkins1", 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: "items": [ 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 0, 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 0 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 1, 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 1 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 2, 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T09:07:02.437 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 2 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: "id": 3, 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: "weight": 6402, 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: "pos": 3 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: "rules": [ 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_id": 0, 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: "rule_name": "replicated_rule", 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: "type": 1, 2024-04-25T09:07:02.438 INFO:tasks.workunit.client.0.smithi102.stdout: "steps": [ 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "take", 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: "item": -1, 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: "item_name": "default" 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "choose_firstn", 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 0, 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: "type": "osd" 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: "op": "emit" 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: ] 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.439 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "tunables": { 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_tries": 0, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_local_fallback_tries": 0, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_total_tries": 50, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_descend_once": 1, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_vary_r": 1, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "chooseleaf_stable": 1, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_descents": 100, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "msr_collision_tries": 100, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "straw_calc_version": 1, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "allowed_bucket_algs": 54, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "profile": "jewel", 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "optimal_tunables": 1, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "legacy_tunables": 0, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "minimum_required_version": "jewel", 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables": 1, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables2": 1, 2024-04-25T09:07:02.440 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v2_rules": 0, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables3": 1, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v3_rules": 0, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v4_buckets": 1, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "require_feature_tunables5": 1, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "has_v5_rules": 0, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "has_msr_rules": 0 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "choose_args": {} 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "fsmap": { 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 1, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "default_fscid": -1, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": { 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "compat": {}, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "ro_compat": {}, 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "incompat": { 2024-04-25T09:07:02.441 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_1": "base v0.20", 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_2": "client writeable ranges", 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_4": "dir inode in separate object", 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_8": "no anchor table", 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_9": "file layout v2", 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_10": "snaprealm v2", 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_11": "minor log segments", 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "feature_flags": { 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "enable_multiple": true, 2024-04-25T09:07:02.442 INFO:tasks.workunit.client.0.smithi102.stdout: "ever_enabled_multiple": true 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "standbys": [], 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "filesystems": [] 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_first_committed": 1, 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "mdsmap_last_committed": 1, 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "auth": { 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 1, 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 6, 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "num_secrets": 4 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 9, 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_active": 9, 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_unknown": 0, 2024-04-25T09:07:02.443 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osd": 4, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_sum": { 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 4096, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 1, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 3, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 1, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 3, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 1, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 1, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 3, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 12, 2024-04-25T09:07:02.444 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 1, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 4, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 3, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 12288, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T09:07:02.445 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 3 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T09:07:02.446 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 0, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 0, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 1, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 1, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 27, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 27, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 0 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum": { 2024-04-25T09:07:02.447 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 27, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 4, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 4, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 4, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 419430400, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 108540, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 1228, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 6, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 107257, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 419321860, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 429496729600, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 429385584640, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T09:07:02.448 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 1257472, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 421264, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 6377, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 109831959 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 4, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [ 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.449 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.450 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 262144 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_sum_by_class": { 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "hdd": { 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "up_from": 0, 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 0, 2024-04-25T09:07:02.451 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pgs": 27, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "num_osds": 4, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_osds": 4, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "num_per_pool_omap_osds": 4, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "kb": 419430400, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used": 108540, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_data": 1228, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_omap": 6, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_used_meta": 107257, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "kb_avail": 419321860, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "statfs": { 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 429496729600, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 429385584640, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 1257472, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 421264, 2024-04-25T09:07:02.452 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 6377, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 109831959 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: "hb_peers": [], 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: "snap_trim_queue_len": 0, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: "num_snap_trimming": 0, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shards_repaired": 4, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: "op_queue_age_hist": { 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: "histogram": [ 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.453 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 0, 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: 1 2024-04-25T09:07:02.454 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: "upper_bound": 262144 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: "perf_stat": { 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ms": 0, 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ms": 0, 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: "commit_latency_ns": 0, 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: "apply_latency_ns": 0 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: "alerts": [], 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: "network_ping_times": [] 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: "pool_stats": [ 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 2, 2024-04-25T09:07:02.455 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 1, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 4096, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 1, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 3, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 1, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 3, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 1, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 1, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 3, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 12, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 1, 2024-04-25T09:07:02.456 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 4, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 3, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 12288, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T09:07:02.457 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 3 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T09:07:02.458 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 12288, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 12288, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 1, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 1, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 3, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 3, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 3 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.459 INFO:tasks.workunit.client.0.smithi102.stdout: "poolid": 1, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg": 8, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "stat_sum": { 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_clones": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_object_copies": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_missing": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_degraded": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_misplaced": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_unfound": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_dirty": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_whiteouts": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_read_kb": 0, 2024-04-25T09:07:02.460 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_write_kb": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_scrub_errors": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_deep_scrub_errors": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_recovered": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_recovered": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_keys_recovered": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_omap": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_kb": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_kb": 0, 2024-04-25T09:07:02.461 INFO:tasks.workunit.client.0.smithi102.stdout: "num_promote": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_high": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "num_flush_mode_low": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_some": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "num_evict_mode_full": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_pinned": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "num_legacy_snapsets": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "num_large_omap_objects": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_manifest": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_bytes": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "num_omap_keys": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "num_objects_repaired": 0 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "store_stats": { 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "total": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "available": 0, 2024-04-25T09:07:02.462 INFO:tasks.workunit.client.0.smithi102.stdout: "internally_reserved": 0, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "allocated": 0, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "data_stored": 0, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed": 0, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_allocated": 0, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "data_compressed_original": 0, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "omap_allocated": 0, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "internal_metadata": 0 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "log_size": 0, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "ondisk_log_size": 0, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "up": 24, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "acting": 24, 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: "num_store_stats": 4 2024-04-25T09:07:02.463 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: "osd_stats": [ 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 25769803857 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 51539607621 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 85899345981 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.464 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 3, 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: "seq": 128849018941 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_state": [ 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "active+recovery_unfound+degraded", 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 1 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: "state": "active+clean", 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: "num": 8 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: "num_pg_by_osd": [ 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.465 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 3, 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 4, 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 7, 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 2, 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 1, 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 5, 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 1, 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 3, 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 6, 2024-04-25T09:07:02.466 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: "osd": 0, 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: "num_primary_pg": 1, 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: "num_acting_pg": 9, 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": [ 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 1, 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": {} 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: { 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: "pool": 2, 2024-04-25T09:07:02.467 INFO:tasks.workunit.client.0.smithi102.stdout: "purged_snaps": {} 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: ], 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "servicemap": { 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "epoch": 6, 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "modified": "2024-04-25T09:05:16.170353+0000", 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "services": {} 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: }, 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_first_committed": 1, 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "mgrstat_last_committed": 152, 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_first_committed": 1, 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "logm_last_committed": 645, 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "paxos": { 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "first_committed": 252, 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "last_committed": 839, 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "last_pn": 0, 2024-04-25T09:07:02.468 INFO:tasks.workunit.client.0.smithi102.stdout: "accepted_pn": 0 2024-04-25T09:07:02.469 INFO:tasks.workunit.client.0.smithi102.stdout: } 2024-04-25T09:07:02.469 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T09:07:02.469 INFO:tasks.workunit.client.0.smithi102.stderr:report 2188453279 2024-04-25T09:07:02.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T09:07:02.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: ceph pg 2.0 query 2024-04-25T09:07:02.470 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:07:02.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: COUNT=3 2024-04-25T09:07:02.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:161: rados_get_data: test 3 = 3 2024-04-25T09:07:02.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:162: rados_get_data: flush_pg_stats 2024-04-25T09:07:02.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:07:02.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:07:02.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:07:02.918 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:07:02.919 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:07:02.919 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:07:02.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:07:02.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:07:02.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:07:03.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803859 2024-04-25T09:07:03.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803859 2024-04-25T09:07:03.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803859' 2024-04-25T09:07:03.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:07:03.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:07:03.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607624 2024-04-25T09:07:03.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607624 2024-04-25T09:07:03.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803859 1-51539607624' 2024-04-25T09:07:03.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:07:03.260 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:07:03.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345983 2024-04-25T09:07:03.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345983 2024-04-25T09:07:03.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803859 1-51539607624 2-85899345983' 2024-04-25T09:07:03.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:07:03.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:07:03.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018943 2024-04-25T09:07:03.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018943 2024-04-25T09:07:03.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803859 1-51539607624 2-85899345983 3-128849018943' 2024-04-25T09:07:03.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:07:03.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803859 2024-04-25T09:07:03.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:07:03.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:07:03.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803859 2024-04-25T09:07:03.542 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:07:03.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803859 2024-04-25T09:07:03.543 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803859 2024-04-25T09:07:03.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803859' 2024-04-25T09:07:03.543 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:07:03.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803857 -lt 25769803859 2024-04-25T09:07:03.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:07:04.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:07:04.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:07:05.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803860 -lt 25769803859 2024-04-25T09:07:05.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:07:05.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607624 2024-04-25T09:07:05.362 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:07:05.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:07:05.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607624 2024-04-25T09:07:05.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:07:05.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607624 2024-04-25T09:07:05.366 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 51539607624 2024-04-25T09:07:05.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607624' 2024-04-25T09:07:05.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:07:05.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607624 -lt 51539607624 2024-04-25T09:07:05.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:07:05.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345983 2024-04-25T09:07:05.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:07:05.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:07:05.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345983 2024-04-25T09:07:05.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:07:05.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345983 2024-04-25T09:07:05.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345983' 2024-04-25T09:07:05.788 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345983 2024-04-25T09:07:05.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:07:06.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345984 -lt 85899345983 2024-04-25T09:07:06.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:07:06.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018943 2024-04-25T09:07:06.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:07:06.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:07:06.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:07:06.195 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018943 2024-04-25T09:07:06.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018943 2024-04-25T09:07:06.196 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018943 2024-04-25T09:07:06.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018943' 2024-04-25T09:07:06.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:07:06.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018944 -lt 128849018943 2024-04-25T09:07:06.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T09:07:06.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:07:07.002 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-04-25T09:07:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: COUNT=4 2024-04-25T09:07:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:164: rados_get_data: test 4 = 4 2024-04-25T09:07:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:177: TEST_rados_get_with_eio: delete_pool pool-rep 2024-04-25T09:07:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-04-25T09:07:07.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-04-25T09:07:07.513 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'pool-rep' does not exist 2024-04-25T09:07:07.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:07:07.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:07.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:07:07.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:07:07.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:07:07.530 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:07:07.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:07:07.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:07:07.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:07:07.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:07:07.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:07:07.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:07:07.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:07:07.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:07:07.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:07:07.845 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:07:07.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:07:07.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:07:07.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:07:07.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:07:07.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:07:07.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:07:07.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:07:07.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:07:07.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:07.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:07:07.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.388535 2024-04-25T09:07:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:07:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:07:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T09:07:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T09:07:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:07.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:07:07.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:07.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:07:07.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:07:07.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:07:07.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:07:07.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:07:07.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:07:07.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:07:07.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:07:07.887 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:07:07.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:07:07.888 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:07:07.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:07:07.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:07:07.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:07:07.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:07:07.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:07:07.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:07:07.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:07:07.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:07:07.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:07:07.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:07:07.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:07:07.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:07.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:07:07.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.388535 2024-04-25T09:07:07.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:07:07.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:07:07.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T09:07:07.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:07:07.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:07.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:07:07.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.388535 2024-04-25T09:07:07.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:07:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:07:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:07:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T09:07:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T09:07:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:07:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:07:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:07:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T09:07:07.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T09:07:08.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:07:08.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:07:08.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:07:08.400 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:07:08.400 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:08.400 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:07:08.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:07:08.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T09:07:08.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:07:08.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:07:08.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:07:08.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:07:08.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:07:08.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:07:08.621 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:07:08.621 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:07:08.621 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:07:08.621 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:07:08.622 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:08.622 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:07:08.622 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-mon.a.asok 2024-04-25T09:07:08.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:07:08.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.388535/ceph-mon.a.asok config get fsid 2024-04-25T09:07:08.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:07:08.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:07:08.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:07:08.732 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:07:08.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:07:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:07:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:07:08.733 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:07:08.734 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:07:08.734 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:08.734 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:07:08.734 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-mon.a.asok 2024-04-25T09:07:08.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:07:08.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.388535/ceph-mon.a.asok config get mon_host 2024-04-25T09:07:08.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T09:07:08.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:08.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:07:08.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:07:08.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:07:08.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T09:07:08.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:07:09.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:07:09.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:07:09.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:07:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:07:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:07:09.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:07:09.763 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:07:09.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:07:09.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T09:07:10.113 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T09:07:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_repair_warning td/osd-rep-recov-eio.sh 2024-04-25T09:07:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:181: TEST_rados_repair_warning: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:10.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: expr 10 + 1 2024-04-25T09:07:10.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: local OBJS=11 2024-04-25T09:07:10.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:184: TEST_rados_repair_warning: setup_osds 4 2024-04-25T09:07:10.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-04-25T09:07:10.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T09:07:10.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T09:07:10.134 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-04-25T09:07:10.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-04-25T09:07:10.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:07:10.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:07:10.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:07:10.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:07:10.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:07:10.139 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:07:10.139 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:10.139 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:07:10.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:07:10.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:07:10.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:07:10.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:07:10.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:07:10.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa5cf26f-cc2c-4c74-b4ee-c594cffea973 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 aa5cf26f-cc2c-4c74-b4ee-c594cffea973' 2024-04-25T09:07:10.141 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 aa5cf26f-cc2c-4c74-b4ee-c594cffea973 2024-04-25T09:07:10.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:07:10.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+HSpmu3NlChAAA7j6QlNfjbHwfk4/vwH/Sg== 2024-04-25T09:07:10.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+HSpmu3NlChAAA7j6QlNfjbHwfk4/vwH/Sg=="}' 2024-04-25T09:07:10.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa5cf26f-cc2c-4c74-b4ee-c594cffea973 -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:07:10.456 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:07:10.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:07:10.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+HSpmu3NlChAAA7j6QlNfjbHwfk4/vwH/Sg== --osd-uuid aa5cf26f-cc2c-4c74-b4ee-c594cffea973 2024-04-25T09:07:10.499 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:10.508+0000 7fe2ece557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:10.525 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:10.532+0000 7fe2ece557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:10.526 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:10.536+0000 7fe2ece557c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:10.527 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:10.536+0000 7fe2ece557c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:07:10.527 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:10.536+0000 7fe2ece557c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:07:10.527 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:10.536+0000 7fe2ece557c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:07:10.565 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:10.572+0000 7fe2ece557c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T09:07:17.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T09:07:17.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:07:17.153 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:07:17.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:07:17.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:07:17.691 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:07:17.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:07:17.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:07:17.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:07:17.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:07:17.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:07:17.721 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:17.732+0000 7fd7d10c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:17.724 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:17.732+0000 7fd7d10c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:17.725 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:17.732+0000 7fd7d10c87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:18.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:07:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:07:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:07:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:07:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:07:18.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:18.188 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:07:18.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:07:18.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:18.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:18.280 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:18.288+0000 7fd7d10c87c0 -1 Falling back to public interface 2024-04-25T09:07:18.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:19.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:19.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:19.613 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:07:19.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:07:19.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:19.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:20.881 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:20.892+0000 7fd7d10c87c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:07:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:21.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:21.032 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:07:21.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:07:21.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:21.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:21.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:22.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:22.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:22.579 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:07:22.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:07:22.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:22.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:23.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:24.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:07:24.007 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:24.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:24.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:25.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:25.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:25.419 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:07:25.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:07:25.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:25.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:25.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:26.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:26.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:26.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:07:26.890 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:07:26.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:26.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:27.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:28.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:28.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:28.339 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:07:28.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:07:28.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:28.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:28.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:29.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:29.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:29.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:07:29.904 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:07:29.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:29.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:30.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:31.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:31.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:31.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:07:31.333 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:07:31.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:31.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:31.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:32.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:32.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:32.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:07:32.795 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:07:32.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:32.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:33.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:34.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:34.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:34.259 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:07:34.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:07:34.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:34.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:34.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:35.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:35.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:35.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:07:35.953 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:07:35.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:35.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:36.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:37.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:37.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:37.635 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:07:37.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:07:37.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:37.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:38.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:39.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:39.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:39.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:07:39.593 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:07:39.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:39.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:40.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:41.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:41.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:41.401 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:07:41.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:07:41.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:41.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:42.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:43.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:43.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:43.038 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:07:43.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:07:43.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:43.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:43.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:44.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:44.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:44.556 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:07:44.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:07:44.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:44.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:45.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:46.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:46.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:46.778 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:07:46.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:07:46.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:46.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:47.475 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1192813660,v1:127.0.0.1:6803/1192813660] [v2:127.0.0.1:6804/1192813660,v1:127.0.0.1:6805/1192813660] exists,up aa5cf26f-cc2c-4c74-b4ee-c594cffea973 2024-04-25T09:07:47.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:07:47.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:07:47.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:07:47.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:07:47.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:07:47.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:07:47.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:07:47.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:07:47.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:07:47.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:07:47.479 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:07:47.479 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:47.479 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:07:47.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:07:47.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:07:47.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T09:07:47.481 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:07:47.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ef1a166e-8eba-457f-b5dd-6f3c4c14eaf3 2024-04-25T09:07:47.482 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 ef1a166e-8eba-457f-b5dd-6f3c4c14eaf3 2024-04-25T09:07:47.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ef1a166e-8eba-457f-b5dd-6f3c4c14eaf3' 2024-04-25T09:07:47.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:07:47.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjHSpm1MO5HhAATLu0zOzUkCPN7GMsBgR2jA== 2024-04-25T09:07:47.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjHSpm1MO5HhAATLu0zOzUkCPN7GMsBgR2jA=="}' 2024-04-25T09:07:47.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ef1a166e-8eba-457f-b5dd-6f3c4c14eaf3 -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:07:48.011 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:07:48.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:07:48.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjHSpm1MO5HhAATLu0zOzUkCPN7GMsBgR2jA== --osd-uuid ef1a166e-8eba-457f-b5dd-6f3c4c14eaf3 2024-04-25T09:07:48.060 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:48.071+0000 7f08eb3467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:48.095 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:48.107+0000 7f08eb3467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:48.097 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:48.107+0000 7f08eb3467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:48.098 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:48.111+0000 7f08eb3467c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:07:48.098 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:48.111+0000 7f08eb3467c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:07:48.098 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:48.111+0000 7f08eb3467c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:07:48.135 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:48.147+0000 7f08eb3467c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T09:07:54.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T09:07:54.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:07:54.804 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:07:54.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:07:54.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:07:55.419 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:07:55.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:07:55.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:07:55.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:07:55.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:07:55.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:07:55.448 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:55.459+0000 7fea4f1687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:55.450 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:55.463+0000 7fea4f1687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:55.451 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:55.463+0000 7fea4f1687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:55.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:07:55.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:07:55.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:07:55.888 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:07:55.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:07:55.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:07:55.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:55.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:07:55.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:55.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:55.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:56.007+0000 7fea4f1687c0 -1 Falling back to public interface 2024-04-25T09:07:56.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:57.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:57.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:57.318 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:07:57.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:07:57.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:57.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:57.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:58.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:58.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:58.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:07:58.720 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:07:58.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:58.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:58.780 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:07:58.791+0000 7fea4f1687c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:07:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:00.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:00.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:00.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:08:00.375 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:08:00.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:00.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:00.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:01.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:01.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:01.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:08:01.980 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:08:01.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:01.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:02.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:03.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:03.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:03.443 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:08:03.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:08:03.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:03.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:04.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:08:05.028 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:08:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:05.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:06.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:06.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:06.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:08:06.543 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:08:06.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:06.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:07.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:08.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:08.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:08.232 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:08:08.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:08:08.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:08.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:08.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:09.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:09.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:09.806 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:08:09.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:08:09.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:09.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:10.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:11.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:11.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:11.237 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:08:11.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:08:11.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:11.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:11.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:12.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:12.694 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:08:12.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:12.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:08:12.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:12.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:13.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:14.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:14.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:14.171 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:08:14.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:08:14.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:14.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:14.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:15.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:15.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:15.686 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:08:15.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:08:15.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:15.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:16.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:17.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:17.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:17.584 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:08:17.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:08:17.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:17.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:18.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:19.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:19.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:19.932 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:08:19.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:08:19.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:19.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:20.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:21.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:21.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:21.857 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:08:21.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:08:21.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:21.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:22.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:23.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:23.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:23.618 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:08:23.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:08:23.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:23.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:24.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:25.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:25.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:25.386 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:08:25.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:08:25.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:25.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:25.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:26.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:26.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:26.820 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T09:08:26.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:08:26.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:26.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:27.509 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/660911133,v1:127.0.0.1:6811/660911133] [v2:127.0.0.1:6812/660911133,v1:127.0.0.1:6813/660911133] exists,up ef1a166e-8eba-457f-b5dd-6f3c4c14eaf3 2024-04-25T09:08:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:08:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:08:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:08:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:08:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:08:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:08:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:08:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:08:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:08:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:08:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:08:27.512 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:08:27.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:08:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2ea02061-d395-4a2c-8592-2dba7a6bd001 2024-04-25T09:08:27.515 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 2ea02061-d395-4a2c-8592-2dba7a6bd001 2024-04-25T09:08:27.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2ea02061-d395-4a2c-8592-2dba7a6bd001' 2024-04-25T09:08:27.515 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:08:27.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCLHSpm9U2oIBAA6u766oLbblnUHMkBgU7MSw== 2024-04-25T09:08:27.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCLHSpm9U2oIBAA6u766oLbblnUHMkBgU7MSw=="}' 2024-04-25T09:08:27.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2ea02061-d395-4a2c-8592-2dba7a6bd001 -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:08:28.021 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:08:28.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:08:28.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCLHSpm9U2oIBAA6u766oLbblnUHMkBgU7MSw== --osd-uuid 2ea02061-d395-4a2c-8592-2dba7a6bd001 2024-04-25T09:08:28.062 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:28.074+0000 7fce05b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:28.065 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:28.074+0000 7fce05b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:28.066 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:28.078+0000 7fce05b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:28.066 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:28.078+0000 7fce05b9d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:08:28.066 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:28.078+0000 7fce05b9d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:08:28.066 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:28.078+0000 7fce05b9d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:08:28.163 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:28.174+0000 7fce05b9d7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:08:35.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:08:35.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:08:35.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:08:35.191 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T09:08:35.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:08:35.749 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:08:35.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:08:35.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:08:35.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:08:35.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:08:35.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:08:35.777 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:35.786+0000 7f26fead27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:35.779 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:35.790+0000 7f26fead27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:35.780 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:35.790+0000 7f26fead27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:36.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:08:36.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:08:36.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:08:36.151 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:08:36.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:08:36.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:08:36.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:36.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:08:36.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:36.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:36.334 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:36.346+0000 7f26fead27c0 -1 Falling back to public interface 2024-04-25T09:08:36.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:37.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:37.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:37.555 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:08:37.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:08:37.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:37.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:37.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:38.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:38.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:38.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:08:38.972 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:08:38.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:38.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:39.108 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:08:39.118+0000 7f26fead27c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:08:39.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:40.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:40.502 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:08:40.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:08:40.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:40.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:41.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:42.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:42.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:42.219 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:08:42.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:08:42.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:42.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:42.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:43.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:43.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:43.712 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:08:43.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:08:43.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:43.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:44.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:45.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:45.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:45.171 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:08:45.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:08:45.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:45.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:45.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:46.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:46.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:46.607 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:08:46.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:08:46.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:46.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:47.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:48.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:48.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:48.131 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:08:48.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:08:48.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:48.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:48.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:49.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:49.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:49.587 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:08:49.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:08:49.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:49.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:49.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:50.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:50.987 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:08:50.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:08:50.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:50.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:51.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:52.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:52.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:52.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:08:52.512 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:08:52.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:52.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:53.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:54.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:54.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:54.109 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:08:54.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:08:54.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:54.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:54.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:55.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:55.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:55.856 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:08:55.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:08:55.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:55.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:57.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:58.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:58.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:58.066 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:08:58.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:08:58.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:58.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:58.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:59.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:59.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:59.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:08:59.889 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:08:59.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:59.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:00.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:01.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:01.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:01.692 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:09:01.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:09:01.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:01.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:02.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:03.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:03.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:03.623 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:09:03.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:09:03.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:03.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:04.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:05.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:05.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:05.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:09:05.935 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:09:05.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:05.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:09:06.687 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1831278463,v1:127.0.0.1:6819/1831278463] [v2:127.0.0.1:6820/1831278463,v1:127.0.0.1:6821/1831278463] exists,up 2ea02061-d395-4a2c-8592-2dba7a6bd001 2024-04-25T09:09:06.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:09:06.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:09:06.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:09:06.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:09:06.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-04-25T09:09:06.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:06.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:09:06.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:09:06.691 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:09:06.692 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:06.692 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:09:06.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:09:06.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:09:06.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:09:06.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:09:06.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:09:06.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:09:06.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:09:06.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:09:06.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:09:06.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:09:06.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:09:06.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:09:06.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:09:06.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-04-25T09:09:06.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:09:06.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=95641fc9-2bac-4544-90b6-37c321f69468 2024-04-25T09:09:06.695 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 95641fc9-2bac-4544-90b6-37c321f69468 2024-04-25T09:09:06.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 95641fc9-2bac-4544-90b6-37c321f69468' 2024-04-25T09:09:06.695 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:09:06.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyHSpmUudmKxAARW18HLxmFnW7G9jwXYISnQ== 2024-04-25T09:09:06.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyHSpmUudmKxAARW18HLxmFnW7G9jwXYISnQ=="}' 2024-04-25T09:09:06.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 95641fc9-2bac-4544-90b6-37c321f69468 -i td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T09:09:07.946 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:09:07.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T09:09:07.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyHSpmUudmKxAARW18HLxmFnW7G9jwXYISnQ== --osd-uuid 95641fc9-2bac-4544-90b6-37c321f69468 2024-04-25T09:09:08.001 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:08.009+0000 7f7511ec37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:08.094 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:08.105+0000 7f7511ec37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:08.096 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:08.105+0000 7f7511ec37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:08.096 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:08.105+0000 7f7511ec37c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:09:08.096 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:08.105+0000 7f7511ec37c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:09:08.096 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:08.105+0000 7f7511ec37c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:09:08.327 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:08.337+0000 7f7511ec37c0 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-04-25T09:09:15.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-04-25T09:09:15.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:09:15.108 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T09:09:15.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T09:09:15.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:09:15.664 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T09:09:15.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T09:09:15.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:09:15.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:09:15.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:09:15.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:09:15.694 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:15.705+0000 7f501aaae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:15.696 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:15.705+0000 7f501aaae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:15.698 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:15.709+0000 7f501aaae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:16.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T09:09:16.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:09:16.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T09:09:16.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:09:16.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:09:16.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:16.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:09:16.126 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:09:16.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:16.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:16.251 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:16.261+0000 7f501aaae7c0 -1 Falling back to public interface 2024-04-25T09:09:16.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:17.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:17.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:17.592 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:09:17.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:09:17.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:17.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:18.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:19.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:19.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:19.006 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:09:19.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:09:19.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:19.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:19.067 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:09:19.077+0000 7f501aaae7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T09:09:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:20.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:20.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:20.425 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:09:20.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:09:20.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:20.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:21.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:22.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:22.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:22.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:09:22.085 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:09:22.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:22.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:22.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:23.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:23.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:23.682 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:09:23.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:09:23.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:23.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:24.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:25.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:25.205 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:09:25.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:09:25.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:25.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:25.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:26.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:26.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:26.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:09:26.619 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:09:26.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:26.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:27.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:28.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:28.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:28.235 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:09:28.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:09:28.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:28.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:28.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:29.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:29.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:29.732 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:09:29.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:09:29.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:29.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:30.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:31.229 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:09:31.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:09:31.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:31.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:31.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:32.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:09:32.782 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:09:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:32.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:33.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:34.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:34.264 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:09:34.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:34.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:09:34.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:34.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:34.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:35.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:35.979 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:09:35.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:09:35.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:35.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:36.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:37.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:37.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:37.684 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:09:37.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:09:37.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:37.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:38.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:39.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:39.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:39.522 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:09:39.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:09:39.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:39.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:40.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:41.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:41.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:41.449 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:09:41.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:09:41.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:41.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:42.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:43.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:43.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:43.403 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:09:43.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:09:43.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:43.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:44.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:45.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:45.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:45.798 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:09:45.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:09:45.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:45.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:46.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:47.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:47.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:47.372 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T09:09:47.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:09:47.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:47.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:09:47.887 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/990156376,v1:127.0.0.1:6827/990156376] [v2:127.0.0.1:6828/990156376,v1:127.0.0.1:6829/990156376] exists,up 95641fc9-2bac-4544-90b6-37c321f69468 2024-04-25T09:09:47.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:09:47.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:09:47.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:09:47.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:09:47.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:09:47.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:09:47.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:09:47.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:09:47.890 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:09:47.890 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:09:47.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:09:47.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:09:47.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:09:47.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:09:47.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:09:47.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:09:47.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:09:47.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:09:47.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:09:48.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:09:48.431 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:09:48.431 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:09:48.431 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:09:48.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:09:48.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:48.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:09:48.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803809 2024-04-25T09:09:48.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803809 2024-04-25T09:09:48.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803809' 2024-04-25T09:09:48.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:48.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:09:48.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574870 2024-04-25T09:09:48.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574870 2024-04-25T09:09:48.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803809 1-55834574870' 2024-04-25T09:09:48.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:48.911 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:09:49.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313232 2024-04-25T09:09:49.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313232 2024-04-25T09:09:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803809 1-55834574870 2-90194313232' 2024-04-25T09:09:49.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:49.177 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:09:49.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018888 2024-04-25T09:09:49.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018888 2024-04-25T09:09:49.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803809 1-55834574870 2-90194313232 3-128849018888' 2024-04-25T09:09:49.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:49.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803809 2024-04-25T09:09:49.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:49.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:09:49.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803809 2024-04-25T09:09:49.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:49.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803809 2024-04-25T09:09:49.340 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803809 2024-04-25T09:09:49.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803809' 2024-04-25T09:09:49.340 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:49.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803807 -lt 25769803809 2024-04-25T09:09:49.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:09:50.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:09:50.842 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:51.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803809 -lt 25769803809 2024-04-25T09:09:51.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:51.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574870 2024-04-25T09:09:51.281 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:51.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:09:51.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574870 2024-04-25T09:09:51.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:51.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574870 2024-04-25T09:09:51.285 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574870 2024-04-25T09:09:51.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574870' 2024-04-25T09:09:51.286 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:09:51.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574870 -lt 55834574870 2024-04-25T09:09:51.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:51.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313232 2024-04-25T09:09:51.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:51.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:09:51.708 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313232 2024-04-25T09:09:51.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:51.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313232 2024-04-25T09:09:51.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313232' 2024-04-25T09:09:51.710 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 90194313232 2024-04-25T09:09:51.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:09:52.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313232 -lt 90194313232 2024-04-25T09:09:52.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:52.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018888 2024-04-25T09:09:52.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:52.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:09:52.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018888 2024-04-25T09:09:52.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:52.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018888 2024-04-25T09:09:52.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018888' 2024-04-25T09:09:52.138 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018888 2024-04-25T09:09:52.138 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:09:52.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018888 -lt 128849018888 2024-04-25T09:09:52.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:09:52.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:09:52.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:09:53.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:09:53.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:09:53.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:09:53.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:09:53.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:09:53.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:09:53.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:09:53.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:09:53.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-25T09:09:53.594 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:09:53.594 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:09:53.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:09:54.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 8 2024-04-25T09:09:54.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 4 '!=' -1 2024-04-25T09:09:54.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:09:54.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=4 2024-04-25T09:09:54.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:09:54.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:09:54.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:09:54.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:09:54.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:09:54.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:09:54.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:09:54.212 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:09:54.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:09:54.213 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:09:54.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:09:54.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:09:54.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:09:54.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:09:55.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:09:55.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:09:55.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:09:55.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:186: TEST_rados_repair_warning: local poolname=pool-rep 2024-04-25T09:09:55.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:187: TEST_rados_repair_warning: create_pool pool-rep 1 1 2024-04-25T09:09:55.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-04-25T09:09:55.728 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'pool-rep' already exists 2024-04-25T09:09:55.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:09:56.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:188: TEST_rados_repair_warning: wait_for_clean 2024-04-25T09:09:56.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:09:56.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:09:56.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:09:56.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:09:56.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:09:56.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:09:56.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:09:56.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:09:56.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:09:56.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:09:56.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:09:56.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:09:56.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:09:56.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:09:56.870 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:09:57.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:09:57.422 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:09:57.422 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:09:57.422 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:09:57.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:09:57.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:57.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:09:57.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803813 2024-04-25T09:09:57.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803813 2024-04-25T09:09:57.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813' 2024-04-25T09:09:57.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:57.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:09:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574874 2024-04-25T09:09:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574874 2024-04-25T09:09:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574874' 2024-04-25T09:09:57.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:57.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:09:57.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313236 2024-04-25T09:09:57.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313236 2024-04-25T09:09:57.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574874 2-90194313236' 2024-04-25T09:09:57.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:57.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:09:58.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018892 2024-04-25T09:09:58.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018892 2024-04-25T09:09:58.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574874 2-90194313236 3-128849018892' 2024-04-25T09:09:58.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:58.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:58.055 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803813 2024-04-25T09:09:58.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:09:58.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803813 2024-04-25T09:09:58.056 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:58.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803813 2024-04-25T09:09:58.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-04-25T09:09:58.057 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803813 2024-04-25T09:09:58.058 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:58.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803810 -lt 25769803813 2024-04-25T09:09:58.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:09:59.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:09:59.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:59.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803813 -lt 25769803813 2024-04-25T09:09:59.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:59.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574874 2024-04-25T09:09:59.852 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:59.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:09:59.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574874 2024-04-25T09:09:59.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:59.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574874 2024-04-25T09:09:59.856 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574874 2024-04-25T09:09:59.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574874' 2024-04-25T09:09:59.857 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:10:00.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574874 -lt 55834574874 2024-04-25T09:10:00.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:00.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313236 2024-04-25T09:10:00.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:00.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:10:00.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313236 2024-04-25T09:10:00.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:00.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313236 2024-04-25T09:10:00.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313236' 2024-04-25T09:10:00.309 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 90194313236 2024-04-25T09:10:00.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:10:00.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313236 -lt 90194313236 2024-04-25T09:10:00.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:00.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018892 2024-04-25T09:10:00.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:00.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:10:00.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018892 2024-04-25T09:10:00.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:00.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018892 2024-04-25T09:10:00.772 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018892 2024-04-25T09:10:00.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018892' 2024-04-25T09:10:00.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:10:01.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018892 -lt 128849018892 2024-04-25T09:10:01.197 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:10:01.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:10:01.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:10:01.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:10:01.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:10:01.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:10:01.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:10:01.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:10:01.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:10:01.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:10:01.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:10:02.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:10:02.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:10:02.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:10:02.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:10:02.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:10:02.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:10:02.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:10:02.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:190: TEST_rados_repair_warning: local poolname=pool-rep 2024-04-25T09:10:02.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:191: TEST_rados_repair_warning: local objbase=obj-warn 2024-04-25T09:10:02.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:192: TEST_rados_repair_warning: local inject=eio 2024-04-25T09:10:02.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: seq 1 11 2024-04-25T09:10:02.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:02.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T09:10:02.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:02.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:10:02.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-1 2024-04-25T09:10:02.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:02.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:10:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:10:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:10:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:10:02.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:10:02.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:02.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:02.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:02.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:02.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:02.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:02.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:02.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T09:10:02.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:02.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:02.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:02.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:02.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:02.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T09:10:02.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:02.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:10:02.837 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:10:02.837 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:03.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:03.323 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:03.323 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:03.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:03.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:03.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:03.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:03.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:03.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:03.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:03.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:03.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:03.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:03.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:03.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:03.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:03.326 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:03.326 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:03.326 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:03.326 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:03.327 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:03.327 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:03.327 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:03.327 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:03.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:03.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:03.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:03.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:03.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:03.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:03.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:03.443 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:03.443 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:03.443 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:03.443 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:03.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:03.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-04-25T09:10:03.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T09:10:03.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:03.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:03.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-04-25T09:10:03.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:03.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:03.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:03.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:03.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:03.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-2 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:10:03.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:10:03.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:03.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:03.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:03.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:03.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:03.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:03.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:03.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T09:10:03.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:03.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:03.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:03.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:03.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:03.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T09:10:03.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:03.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T09:10:03.783 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T09:10:03.783 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:04.190 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:04.193 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:04.193 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:04.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:04.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:04.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:04.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:04.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:04.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:04.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:04.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:04.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:04.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:04.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:04.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:04.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:04.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:04.196 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:04.196 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:04.196 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:04.196 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:04.196 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:04.196 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:04.196 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:04.196 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:04.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:04.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:04.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:04.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:04.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:04.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:04.315 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:04.315 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:04.315 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:04.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:04.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:04.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-04-25T09:10:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T09:10:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-04-25T09:10:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:04.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:04.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:04.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-3 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:10:04.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:04.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:10:04.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:10:04.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:04.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:04.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:04.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:04.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:04.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:04.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:04.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T09:10:04.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:04.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:04.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:04.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:04.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:04.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T09:10:04.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:04.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T09:10:04.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T09:10:04.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:05.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:05.108 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:05.108 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:05.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:05.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:05.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:05.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:05.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:05.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:05.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:05.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:05.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:05.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:05.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:05.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:05.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:05.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:05.111 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:05.111 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:05.111 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:05.111 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:05.111 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:05.112 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:05.112 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:05.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:05.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:05.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:05.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:05.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:05.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:05.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:05.227 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:05.227 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:05.227 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:05.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:05.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:05.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-04-25T09:10:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T09:10:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-04-25T09:10:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:05.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:05.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:05.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:05.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-4 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:10:05.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:10:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:05.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:05.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T09:10:05.582 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:05.582 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T09:10:05.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:05.975 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:05.975 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:05.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:05.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:05.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:05.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:05.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:05.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:05.976 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:05.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:05.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:05.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:05.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:05.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:05.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:05.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:05.978 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:05.978 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:05.978 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:05.978 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:05.978 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:05.978 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:05.978 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:05.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:06.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:06.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:06.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:06.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:06.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:06.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:06.086 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:06.086 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:06.086 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:06.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:06.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:06.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-04-25T09:10:06.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T09:10:06.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:06.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:06.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-04-25T09:10:06.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:06.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:06.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:06.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:06.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:06.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:06.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-5 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:10:06.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:10:06.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:06.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:06.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:06.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:06.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:06.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:06.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:06.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T09:10:06.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:06.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:06.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:06.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:06.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:06.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T09:10:06.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:06.459 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T09:10:06.459 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T09:10:06.459 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:06.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:06.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:06.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:06.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:06.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:06.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:06.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:06.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:06.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:06.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:06.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:06.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:06.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:06.871 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:06.871 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:06.871 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:06.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:06.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:06.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:06.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:06.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:06.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:06.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:06.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:06.993 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:06.993 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:06.994 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:06.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:06.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:06.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-04-25T09:10:07.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T09:10:07.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:07.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:07.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-04-25T09:10:07.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:07.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:07.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:07.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:07.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:07.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:07.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:07.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T09:10:07.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:07.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:10:07.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-6 2024-04-25T09:10:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:10:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:10:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:10:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:10:07.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:10:07.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:07.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:07.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:07.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:07.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:07.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:07.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:07.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T09:10:07.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:07.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:07.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:07.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:07.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:07.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T09:10:07.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:07.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T09:10:07.395 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T09:10:07.395 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:07.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:07.787 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:07.787 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:07.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:07.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:07.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:07.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:07.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:07.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:07.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:07.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:07.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:07.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:07.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:07.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:07.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:07.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:07.790 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:07.790 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:07.790 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:07.791 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:07.791 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:07.791 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:07.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:07.791 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:07.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:07.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:07.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:07.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:07.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:07.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:07.907 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:07.907 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:07.907 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:07.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:07.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:07.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-04-25T09:10:08.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T09:10:08.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:08.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:08.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-04-25T09:10:08.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:08.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:08.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:08.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:08.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:08.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:08.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:08.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-7 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:10:08.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:10:08.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:08.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:08.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:08.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:08.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T09:10:08.316 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T09:10:08.316 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:08.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:08.741 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:08.741 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:08.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:08.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:08.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:08.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:08.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:08.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:08.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:08.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:08.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:08.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:08.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:08.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:08.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:08.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:08.743 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:08.743 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:08.743 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:08.744 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:08.744 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:08.744 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:08.745 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:08.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:08.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:08.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:08.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:08.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:08.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:08.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:08.862 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:08.862 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:08.862 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:08.862 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:08.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:08.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-04-25T09:10:08.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T09:10:08.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:08.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:08.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-04-25T09:10:08.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:08.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:08.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:08.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:09.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:09.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T09:10:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:10:09.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-8 2024-04-25T09:10:09.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:09.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:10:09.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:09.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:10:09.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:09.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:10:09.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:09.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:10:09.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:10:09.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T09:10:09.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:09.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T09:10:09.211 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T09:10:09.211 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:09.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:09.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:09.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:09.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:09.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:09.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:09.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:09.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:09.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:09.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:09.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:09.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:09.674 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:09.674 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:09.674 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:09.674 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:09.674 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:09.674 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:09.675 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:09.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:09.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:09.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:09.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:09.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:09.783 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:09.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:09.784 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:09.784 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:09.784 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:09.784 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:09.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:09.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-04-25T09:10:09.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T09:10:09.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:09.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:09.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-04-25T09:10:09.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:09.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:09.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:09.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:09.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:09.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:09.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:09.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-9 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:10:09.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:10:10.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:10.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:10.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:10.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T09:10:10.132 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T09:10:10.132 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:10.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:10.585 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:10.585 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:10.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:10.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:10.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:10.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:10.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:10.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:10.586 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:10.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:10.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:10.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:10.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:10.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:10.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:10.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:10.589 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:10.589 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:10.589 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:10.589 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:10.590 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:10.590 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:10.590 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:10.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:10.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:10.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:10.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:10.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:10.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:10.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:10.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:10.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:10.707 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:10.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:10.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:10.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-04-25T09:10:10.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T09:10:10.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:10.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:10.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-04-25T09:10:10.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:10.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:10.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:10.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:10.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:10.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:10.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:10.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-10 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:10:10.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:10:11.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:11.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T09:10:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T09:10:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:11.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:11.471 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:11.471 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:11.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:11.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:11.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:11.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:11.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:11.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:11.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:11.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:11.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:11.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:11.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:11.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:11.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:11.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:11.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:11.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:11.474 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:11.475 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:11.475 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:11.475 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:11.475 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:11.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:11.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:11.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:11.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:11.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:11.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:11.584 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:11.585 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:11.585 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:11.585 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:11.585 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:11.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:11.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-04-25T09:10:11.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T09:10:11.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:11.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:11.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-04-25T09:10:11.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:11.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:11.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:11.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:11.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:11.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-11 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:10:11.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:10:11.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:11.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:11.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:11.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:11.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:11.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:11.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:11.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T09:10:11.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:11.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:11.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:11.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:11.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:11.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T09:10:11.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:11.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T09:10:11.955 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T09:10:11.955 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:12.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:12.389 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:12.389 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:12.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:12.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:12.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:12.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:12.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:12.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:12.390 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:12.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:12.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:12.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:12.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:12.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:12.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:12.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:12.391 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:12.392 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:12.392 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:12.392 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:12.392 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:12.392 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:12.393 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:12.393 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:12.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:12.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:12.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:12.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:12.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:12.516 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:12.516 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:12.516 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:12.516 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:12.517 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:12.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:12.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-04-25T09:10:12.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T09:10:12.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:12.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:12.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-04-25T09:10:12.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:12.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:12.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:12.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:12.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:12.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:12.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: get_pg pool-rep obj-warn-1 2024-04-25T09:10:12.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-04-25T09:10:12.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-warn-1 2024-04-25T09:10:12.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:10:12.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T09:10:13.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: local pgid=2.0 2024-04-25T09:10:13.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: get_osds pool-rep obj-warn-1 2024-04-25T09:10:13.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:13.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:10:13.107 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:10:13.108 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:13.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:13.549 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:13.549 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:13.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:13.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: object_osds=('3' '1' '0') 2024-04-25T09:10:13.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: local object_osds 2024-04-25T09:10:13.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:203: TEST_rados_repair_warning: local primary=3 2024-04-25T09:10:13.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:204: TEST_rados_repair_warning: local bad_peer=1 2024-04-25T09:10:13.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:206: TEST_rados_repair_warning: wait_for_clean 2024-04-25T09:10:13.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:10:13.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:10:13.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:10:13.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:10:13.551 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:10:13.551 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:10:13.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:10:13.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:10:13.551 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:10:13.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:10:13.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:10:13.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:10:13.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:10:13.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:10:13.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:10:14.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:10:14.042 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:14.042 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:10:14.042 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:10:14.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:10:14.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:14.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:10:14.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803818 2024-04-25T09:10:14.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803818 2024-04-25T09:10:14.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803818' 2024-04-25T09:10:14.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:14.218 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:10:14.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574879 2024-04-25T09:10:14.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574879 2024-04-25T09:10:14.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803818 1-55834574879' 2024-04-25T09:10:14.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:14.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:10:14.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313241 2024-04-25T09:10:14.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313241 2024-04-25T09:10:14.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803818 1-55834574879 2-90194313241' 2024-04-25T09:10:14.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:14.514 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:10:14.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018897 2024-04-25T09:10:14.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018897 2024-04-25T09:10:14.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803818 1-55834574879 2-90194313241 3-128849018897' 2024-04-25T09:10:14.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:14.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803818 2024-04-25T09:10:14.651 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:14.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:10:14.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803818 2024-04-25T09:10:14.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:14.653 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803818 2024-04-25T09:10:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803818 2024-04-25T09:10:14.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803818' 2024-04-25T09:10:14.654 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:10:15.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803818 -lt 25769803818 2024-04-25T09:10:15.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:15.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574879 2024-04-25T09:10:15.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:15.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:10:15.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574879 2024-04-25T09:10:15.045 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:15.046 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574879 2024-04-25T09:10:15.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574879 2024-04-25T09:10:15.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574879' 2024-04-25T09:10:15.046 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:10:15.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574877 -lt 55834574879 2024-04-25T09:10:15.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:10:16.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:10:16.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:10:16.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574879 -lt 55834574879 2024-04-25T09:10:16.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:16.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313241 2024-04-25T09:10:16.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:16.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:10:16.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313241 2024-04-25T09:10:16.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:16.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313241 2024-04-25T09:10:16.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313241' 2024-04-25T09:10:16.868 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 90194313241 2024-04-25T09:10:16.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:10:17.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313241 -lt 90194313241 2024-04-25T09:10:17.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:17.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018897 2024-04-25T09:10:17.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:17.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:10:17.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018897 2024-04-25T09:10:17.293 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:17.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018897 2024-04-25T09:10:17.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018897' 2024-04-25T09:10:17.295 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018897 2024-04-25T09:10:17.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:10:17.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018897 -lt 128849018897 2024-04-25T09:10:17.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:10:17.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:10:17.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:10:18.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:10:18.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:10:18.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:10:18.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:10:18.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:10:18.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:10:18.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:10:18.331 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:10:18.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:10:18.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:10:18.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:10:18.717 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:10:19.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:10:19.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:10:19.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:10:19.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: ceph pg 2.0 query 2024-04-25T09:10:19.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:10:19.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: COUNT=11 2024-04-25T09:10:19.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:208: TEST_rados_repair_warning: test 11 = 11 2024-04-25T09:10:19.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:209: TEST_rados_repair_warning: flush_pg_stats 2024-04-25T09:10:19.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:10:19.431 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:10:19.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:10:19.835 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:19.835 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:10:19.835 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:10:19.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:10:19.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:19.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:10:19.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803821 2024-04-25T09:10:19.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803821 2024-04-25T09:10:19.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803821' 2024-04-25T09:10:19.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:19.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:10:20.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574882 2024-04-25T09:10:20.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574882 2024-04-25T09:10:20.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803821 1-55834574882' 2024-04-25T09:10:20.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:20.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:10:20.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313244 2024-04-25T09:10:20.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313244 2024-04-25T09:10:20.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803821 1-55834574882 2-90194313244' 2024-04-25T09:10:20.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:20.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:10:20.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018900 2024-04-25T09:10:20.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018900 2024-04-25T09:10:20.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803821 1-55834574882 2-90194313244 3-128849018900' 2024-04-25T09:10:20.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:20.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803821 2024-04-25T09:10:20.414 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:20.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:10:20.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803821 2024-04-25T09:10:20.416 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:20.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803821 2024-04-25T09:10:20.417 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803821 2024-04-25T09:10:20.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803821' 2024-04-25T09:10:20.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:10:20.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803821 -lt 25769803821 2024-04-25T09:10:20.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:20.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574882 2024-04-25T09:10:20.829 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:20.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:10:20.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574882 2024-04-25T09:10:20.831 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:20.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574882 2024-04-25T09:10:20.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574882' 2024-04-25T09:10:20.832 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574882 2024-04-25T09:10:20.833 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:10:21.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574882 -lt 55834574882 2024-04-25T09:10:21.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:21.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313244 2024-04-25T09:10:21.252 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:21.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:10:21.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313244 2024-04-25T09:10:21.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:21.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313244 2024-04-25T09:10:21.255 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 90194313244 2024-04-25T09:10:21.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313244' 2024-04-25T09:10:21.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:10:21.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313244 -lt 90194313244 2024-04-25T09:10:21.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:21.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018900 2024-04-25T09:10:21.701 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:21.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:10:21.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018900 2024-04-25T09:10:21.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:21.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018900 2024-04-25T09:10:21.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018900' 2024-04-25T09:10:21.704 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018900 2024-04-25T09:10:21.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:10:22.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018898 -lt 128849018900 2024-04-25T09:10:22.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:10:23.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:10:23.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:10:23.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018900 -lt 128849018900 2024-04-25T09:10:23.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-04-25T09:10:23.539 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:10:23.957 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-04-25T09:10:23.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: COUNT=11 2024-04-25T09:10:23.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:211: TEST_rados_repair_warning: test 11 = 11 2024-04-25T09:10:23.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: ceph health 2024-04-25T09:10:23.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-04-25T09:10:24.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: ceph health detail 2024-04-25T09:10:24.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: grep -q 'osd.3 had 11 reads repaired' 2024-04-25T09:10:25.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:216: TEST_rados_repair_warning: ceph health mute OSD_TOO_MANY_REPAIRS 2024-04-25T09:10:25.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:217: TEST_rados_repair_warning: set -o pipefail 2024-04-25T09:10:25.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: ceph health 2024-04-25T09:10:25.805 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-04-25T09:10:26.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:220: TEST_rados_repair_warning: set +o pipefail 2024-04-25T09:10:26.279 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: seq 1 11 2024-04-25T09:10:26.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:26.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:26.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:26.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T09:10:26.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:26.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:10:26.283 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:10:26.283 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:26.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:26.742 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:26.742 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:26.742 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:26.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:26.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:26.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:26.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:26.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:26.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:26.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:26.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:26.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:26.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:26.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:26.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:26.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:26.745 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:26.745 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:26.745 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:26.746 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:26.746 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:26.746 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:26.746 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:26.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:26.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:26.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:26.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:26.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:26.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:26.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:26.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:26.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:26.871 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:26.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:26.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:26.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-04-25T09:10:26.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:26.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:26.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:26.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:26.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:26.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:26.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:26.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T09:10:26.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:26.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:26.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:26.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:10:26.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:26.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T09:10:26.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:26.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:10:26.990 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:10:26.990 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:27.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:27.398 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:27.398 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:27.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:27.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:27.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:27.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:10:27.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:27.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:27.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:10:27.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:27.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:10:27.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:27.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:10:27.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:27.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:27.400 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:27.401 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:10:27.401 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:27.401 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:27.402 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:27.402 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:27.402 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:27.402 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:27.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:27.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:27.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:27.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:27.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:10:27.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:27.510 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:27.511 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:27.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-warn-1 2024-04-25T09:10:27.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:10:27.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:27.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:27.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:10:27.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:10:27.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:27.637 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:27.637 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:27.637 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:27.637 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:27.638 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:27.638 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:27.638 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:27.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:10:27.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:27.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T09:10:27.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:27.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:27.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-04-25T09:10:27.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:27.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:27.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:27.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:27.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:27.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:27.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:27.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:27.844 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T09:10:27.845 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T09:10:27.845 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:28.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:28.311 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:28.311 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:28.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:28.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:28.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:28.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:28.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:28.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:28.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:28.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:28.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:28.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:28.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:28.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:28.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:28.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:28.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:28.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:28.314 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:28.315 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:28.315 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:28.315 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:28.315 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:28.315 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:28.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:28.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:28.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:28.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:28.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:28.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:28.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:28.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:28.436 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:28.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:28.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:28.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-04-25T09:10:28.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:10:28.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:28.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T09:10:28.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:28.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T09:10:28.555 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T09:10:28.555 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:28.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:28.995 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:28.995 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:28.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:28.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:28.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:28.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:10:28.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:28.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:28.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:10:28.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:28.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:10:28.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:28.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:10:28.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:28.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:28.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:28.999 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:10:28.999 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:28.999 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:28.999 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:28.999 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:28.999 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:28.999 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:29.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:29.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:29.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:29.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:29.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:10:29.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:29.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:29.117 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:29.117 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:29.117 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:29.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:29.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:29.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-warn-2 2024-04-25T09:10:29.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:10:29.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:29.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:29.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:10:29.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:10:29.224 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:29.224 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:29.224 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:29.225 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:29.225 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:29.225 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:29.226 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:29.226 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:29.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:10:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T09:10:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-04-25T09:10:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:29.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:29.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:29.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:29.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:29.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:29.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T09:10:29.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:29.392 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T09:10:29.392 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T09:10:29.392 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:29.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:29.788 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:29.788 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:29.788 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:29.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:29.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:29.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:29.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:29.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:29.789 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:29.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:29.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:29.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:29.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:29.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:29.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:29.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:29.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:29.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:29.791 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:29.792 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:29.792 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:29.792 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:29.792 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:29.792 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:29.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:29.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:29.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:29.913 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:29.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:29.914 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:29.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:29.914 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:29.915 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:29.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:29.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:29.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-04-25T09:10:30.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:30.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:30.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:30.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:30.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T09:10:30.036 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T09:10:30.037 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:30.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:30.478 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:30.478 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:30.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:30.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:30.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:30.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:10:30.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:30.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:30.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:10:30.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:30.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:10:30.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:30.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:10:30.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:30.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:30.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:30.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:10:30.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:30.481 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:30.482 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:30.482 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:30.482 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:30.482 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:30.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:30.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:30.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:30.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:30.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:10:30.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:30.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:30.597 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:30.598 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:30.598 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:30.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:30.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:30.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-warn-3 2024-04-25T09:10:30.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:10:30.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:30.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:30.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:10:30.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:10:30.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:30.705 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:30.705 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:30.705 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:30.705 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:30.705 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:30.706 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:30.706 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:30.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:10:30.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:30.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T09:10:30.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:30.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:30.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-04-25T09:10:30.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:30.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:30.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:30.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:30.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:30.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:30.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:30.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:30.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:30.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:30.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:30.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T09:10:30.875 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T09:10:30.875 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:31.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:31.270 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:31.270 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:31.270 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:31.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:31.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:31.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:31.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:31.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:31.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:31.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:31.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:31.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:31.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:31.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:31.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:31.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:31.273 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:31.273 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:31.273 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:31.273 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:31.273 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:31.273 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:31.273 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:31.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:31.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:31.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:31.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:31.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:31.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:31.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:31.383 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:31.383 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:31.383 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:31.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:31.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:31.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-04-25T09:10:31.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:31.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T09:10:31.494 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T09:10:31.494 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:31.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:31.942 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:31.942 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:31.942 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:31.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:31.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:31.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:10:31.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:31.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:31.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:10:31.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:31.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:10:31.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:31.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:10:31.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:31.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:31.945 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:31.945 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:10:31.945 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:31.945 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:31.946 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:31.946 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:31.946 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:31.946 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:31.946 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:32.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:32.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:32.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:32.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:10:32.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:32.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:32.063 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:32.063 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:32.063 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:32.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:32.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:32.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-warn-4 2024-04-25T09:10:32.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:10:32.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:32.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:32.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:10:32.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:10:32.173 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:32.174 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:32.174 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:32.174 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:32.175 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:32.175 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:32.175 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:32.175 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:32.175 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:10:32.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:32.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T09:10:32.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:32.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:32.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-04-25T09:10:32.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:32.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:32.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:32.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:32.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:32.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:32.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:32.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:32.368 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T09:10:32.369 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T09:10:32.369 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:32.765 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:32.766 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:32.766 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:32.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:32.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:32.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:32.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:32.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:32.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:32.767 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:32.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:32.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:32.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:32.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:32.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:32.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:32.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:32.769 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:32.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:32.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:32.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:32.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:32.890 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:32.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:32.891 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:32.891 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:32.891 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:32.892 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:32.892 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:32.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:32.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-04-25T09:10:33.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:33.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:33.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:33.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:33.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:33.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:33.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:33.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T09:10:33.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:33.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:33.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:33.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:10:33.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:33.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T09:10:33.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:33.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T09:10:33.020 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T09:10:33.020 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:33.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:33.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:10:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:33.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:33.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:10:33.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:33.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:10:33.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:33.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:10:33.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:33.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:33.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:33.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:10:33.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:33.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:33.451 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:33.451 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:33.451 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:33.451 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:33.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:33.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:33.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:33.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:10:33.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:33.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:33.569 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:33.570 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:33.570 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:33.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:33.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:33.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-warn-5 2024-04-25T09:10:33.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:10:33.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:33.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:33.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:10:33.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:10:33.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:33.689 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:33.690 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:33.690 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:33.690 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:33.690 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:33.690 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:33.690 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:33.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:10:33.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:33.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T09:10:33.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:33.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:33.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-04-25T09:10:33.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:33.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:33.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:33.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:33.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:33.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:33.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:33.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T09:10:33.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:33.867 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T09:10:33.867 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T09:10:33.867 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:34.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:34.280 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:34.280 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:34.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:34.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:34.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:34.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:34.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:34.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:34.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:34.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:34.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:34.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:34.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:34.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:34.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:34.282 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:34.282 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:34.282 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:34.283 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:34.283 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:34.283 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:34.283 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:34.283 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:34.283 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:34.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:34.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:34.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:34.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:34.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:34.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:34.407 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:34.407 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:34.407 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:34.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:34.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:34.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-04-25T09:10:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:34.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T09:10:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T09:10:34.529 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:34.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:34.934 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:34.934 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:34.934 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:34.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:34.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:34.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:10:34.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:34.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:34.935 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:10:34.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:34.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:10:34.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:34.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:10:34.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:34.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:34.936 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:34.936 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:10:34.936 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:34.937 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:34.937 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:34.937 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:34.937 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:34.938 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:34.938 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:35.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:35.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:35.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:35.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:10:35.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:35.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:35.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:35.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:35.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:35.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:35.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:35.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-warn-6 2024-04-25T09:10:35.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:10:35.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:35.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:35.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:10:35.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:10:35.162 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:35.162 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:35.162 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:35.162 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:35.163 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:35.163 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:35.163 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:35.163 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:35.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:10:35.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:35.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T09:10:35.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:35.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:35.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-04-25T09:10:35.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:35.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:35.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:35.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:35.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:35.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:35.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T09:10:35.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:35.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:35.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:35.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:35.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:35.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T09:10:35.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:35.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T09:10:35.335 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:35.335 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T09:10:35.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:35.729 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:35.729 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:35.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:35.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:35.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:35.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:35.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:35.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:35.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:35.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:35.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:35.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:35.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:35.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:35.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:35.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:35.732 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:35.733 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:35.733 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:35.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:35.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:35.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:35.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:35.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:35.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:35.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:35.848 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:35.848 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:35.848 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:35.848 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:35.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:35.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-04-25T09:10:35.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:35.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:35.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:35.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:35.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:35.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:35.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:35.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T09:10:35.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:35.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:35.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:35.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:10:35.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:35.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T09:10:35.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:35.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T09:10:35.971 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T09:10:35.971 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:36.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:36.405 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:36.405 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:36.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:36.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:36.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:36.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:10:36.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:36.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:36.406 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:10:36.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:36.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:10:36.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:36.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:10:36.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:36.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:36.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:36.409 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:10:36.409 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:36.409 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:36.409 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:36.409 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:36.409 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:36.409 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:36.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:36.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:36.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:36.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:36.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:10:36.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:36.520 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:36.521 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:36.521 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:36.521 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:36.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:36.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:36.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-warn-7 2024-04-25T09:10:36.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:10:36.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:36.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:36.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:10:36.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:10:36.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:36.630 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:36.630 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:36.630 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:36.630 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:36.630 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:36.630 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:36.630 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:36.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:10:36.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:36.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T09:10:36.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:36.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:36.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-04-25T09:10:36.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:36.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:36.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:36.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:36.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:36.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:36.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T09:10:36.799 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:37.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:37.235 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:37.235 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:37.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:37.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:37.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:37.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:37.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:37.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:37.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:37.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:37.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:37.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:37.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:37.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:37.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:37.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:37.239 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:37.239 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:37.239 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:37.239 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:37.239 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:37.239 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:37.239 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:37.239 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:37.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:37.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:37.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:37.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:37.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:37.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:37.360 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:37.360 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:37.360 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:37.360 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:37.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:37.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-04-25T09:10:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T09:10:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:37.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:37.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:37.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:10:37.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:37.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T09:10:37.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:37.482 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T09:10:37.483 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T09:10:37.483 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:37.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:37.894 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:37.895 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:37.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:37.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:37.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:37.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:10:37.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:37.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:37.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:10:37.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:37.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:10:37.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:37.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:10:37.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:37.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:37.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:37.898 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:10:37.898 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:37.898 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:37.898 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:37.898 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:37.898 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:37.898 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:37.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:38.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:38.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:38.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:38.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:10:38.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:38.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:38.007 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:38.008 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:38.008 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:38.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:38.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:38.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-warn-8 2024-04-25T09:10:38.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:10:38.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:38.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:38.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:10:38.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:10:38.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:38.124 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:38.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:10:38.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:38.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T09:10:38.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:38.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:38.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-04-25T09:10:38.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:38.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:38.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:38.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:38.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:38.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:38.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:38.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:38.295 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T09:10:38.296 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T09:10:38.296 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:38.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:38.744 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:38.744 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:38.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:38.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:38.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:38.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:38.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:38.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:38.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:38.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:38.747 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:38.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:38.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:38.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:38.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:38.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:38.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:38.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:38.865 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:38.865 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:38.865 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:38.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:38.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:38.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-04-25T09:10:38.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:39.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T09:10:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:10:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:39.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T09:10:39.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:39.007 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T09:10:39.007 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:39.007 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T09:10:39.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:39.374 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:39.374 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:39.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:10:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:39.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:39.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:10:39.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:39.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:10:39.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:39.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:10:39.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:39.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:39.378 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:39.378 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:10:39.378 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:39.378 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:39.379 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:39.379 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:39.379 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:39.379 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:39.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:39.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:39.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:39.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:39.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:10:39.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:39.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:39.496 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:39.496 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:39.496 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:39.496 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:39.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:39.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-warn-9 2024-04-25T09:10:39.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:10:39.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:39.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:39.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:10:39.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:10:39.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:39.613 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:39.613 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:39.613 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:39.613 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:39.613 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:39.613 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:39.614 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:39.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:10:39.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:39.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T09:10:39.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:39.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:39.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-04-25T09:10:39.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:39.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:39.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:39.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:39.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:39.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:39.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:39.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:39.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:39.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:39.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:39.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:39.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:39.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:39.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T09:10:39.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:39.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:39.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:39.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:39.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:39.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T09:10:39.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:39.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T09:10:39.782 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T09:10:39.782 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:40.186 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:40.187 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:40.188 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:40.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:40.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:40.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:40.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:40.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:40.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:40.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:40.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:40.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:40.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:40.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:40.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:40.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:40.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:40.191 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:40.191 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:40.191 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:40.191 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:40.191 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:40.192 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:40.192 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:40.192 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:40.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:40.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:40.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:40.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:40.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:40.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:40.312 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:40.312 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:40.313 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:40.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:40.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:40.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-04-25T09:10:40.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:40.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:40.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:40.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:40.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:40.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:40.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:40.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T09:10:40.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:40.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:40.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:40.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:10:40.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:40.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T09:10:40.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:40.432 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T09:10:40.432 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T09:10:40.432 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:40.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:40.835 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:40.835 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:40.835 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:40.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:40.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:40.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:10:40.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:40.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:40.836 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:10:40.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:40.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:10:40.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:40.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:10:40.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:40.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:40.838 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:40.838 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:10:40.838 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:40.838 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:40.839 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:40.839 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:40.839 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:40.839 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:40.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:40.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:40.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:40.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:40.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:10:40.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:40.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:40.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:40.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:40.949 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:40.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:40.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:40.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-warn-10 2024-04-25T09:10:41.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:10:41.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:41.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:41.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:10:41.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:10:41.066 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:41.066 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:41.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:41.066 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:41.066 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:41.066 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:41.067 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:41.067 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:41.067 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:10:41.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:41.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T09:10:41.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:41.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:41.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-04-25T09:10:41.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:41.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:41.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:41.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:41.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:41.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:41.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:10:41.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:41.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:41.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:41.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:41.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:41.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:41.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:41.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T09:10:41.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:41.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:41.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:41.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:10:41.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:41.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T09:10:41.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:41.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T09:10:41.223 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T09:10:41.223 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:41.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:41.626 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:41.626 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:41.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:41.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:41.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:41.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:10:41.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:41.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:41.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:10:41.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:41.629 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:41.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:41.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:41.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:41.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:41.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:10:41.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:41.736 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:41.737 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:41.737 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:41.737 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:41.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:41.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:41.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-04-25T09:10:41.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:41.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:10:41.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:10:41.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:10:41.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:10:41.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:10:41.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:10:41.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T09:10:41.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:10:41.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:41.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:10:41.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:10:41.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:10:41.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T09:10:41.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:10:41.854 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T09:10:41.855 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T09:10:41.855 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:10:42.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:10:42.276 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:42.276 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:10:42.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:10:42.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:10:42.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:10:42.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:10:42.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:10:42.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:10:42.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:10:42.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:10:42.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:10:42.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:42.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:10:42.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:10:42.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:10:42.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:42.280 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:10:42.280 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:42.280 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:42.280 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:42.280 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:42.280 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:42.280 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:42.280 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:10:42.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:42.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:10:42.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:10:42.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:10:42.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:10:42.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:10:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:42.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:10:42.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:10:42.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr pool-rep obj-warn-11 2024-04-25T09:10:42.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:10:42.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:10:42.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:10:42.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:10:42.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:10:42.527 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:10:42.527 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:10:42.527 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:10:42.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:10:42.528 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:42.528 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:42.528 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:42.528 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.3.asok 2024-04-25T09:10:42.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:10:42.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:10:42.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T09:10:42.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:42.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:10:42.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-04-25T09:10:42.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:10:42.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:10:42.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:10:42.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:42.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:42.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:10:42.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:231: TEST_rados_repair_warning: wait_for_clean 2024-04-25T09:10:42.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:10:42.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:10:42.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:10:42.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:10:42.706 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:10:42.706 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:10:42.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:10:42.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:10:42.706 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:10:42.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:10:42.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:10:42.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:10:42.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:10:42.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:10:42.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:10:43.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:10:43.256 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:43.256 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:10:43.256 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:10:43.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:10:43.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:43.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:10:43.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803828 2024-04-25T09:10:43.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803828 2024-04-25T09:10:43.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803828' 2024-04-25T09:10:43.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:43.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:10:43.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574889 2024-04-25T09:10:43.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574889 2024-04-25T09:10:43.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803828 1-55834574889' 2024-04-25T09:10:43.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:43.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:10:43.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313251 2024-04-25T09:10:43.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313251 2024-04-25T09:10:43.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803828 1-55834574889 2-90194313251' 2024-04-25T09:10:43.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:43.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:10:43.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018907 2024-04-25T09:10:43.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018907 2024-04-25T09:10:43.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803828 1-55834574889 2-90194313251 3-128849018907' 2024-04-25T09:10:43.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:43.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803828 2024-04-25T09:10:43.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:43.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:10:43.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803828 2024-04-25T09:10:43.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:43.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803828 2024-04-25T09:10:43.883 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803828 2024-04-25T09:10:43.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803828' 2024-04-25T09:10:43.884 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:10:44.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803825 -lt 25769803828 2024-04-25T09:10:44.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:10:45.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:10:45.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:10:45.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803828 -lt 25769803828 2024-04-25T09:10:45.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:45.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574889 2024-04-25T09:10:45.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:45.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:10:45.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574889 2024-04-25T09:10:45.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:45.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574889 2024-04-25T09:10:45.690 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574889 2024-04-25T09:10:45.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574889' 2024-04-25T09:10:45.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:10:46.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574889 -lt 55834574889 2024-04-25T09:10:46.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:46.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313251 2024-04-25T09:10:46.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:46.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:10:46.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313251 2024-04-25T09:10:46.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:46.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313251 2024-04-25T09:10:46.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313251' 2024-04-25T09:10:46.113 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 90194313251 2024-04-25T09:10:46.113 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:10:46.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313251 -lt 90194313251 2024-04-25T09:10:46.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:46.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018907 2024-04-25T09:10:46.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:46.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:10:46.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018907 2024-04-25T09:10:46.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:46.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018907 2024-04-25T09:10:46.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018907' 2024-04-25T09:10:46.598 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018907 2024-04-25T09:10:46.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:10:47.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018907 -lt 128849018907 2024-04-25T09:10:47.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:10:47.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:10:47.026 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:10:47.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:10:47.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:10:47.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:10:47.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:10:47.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:10:47.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:10:47.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:10:47.618 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:10:48.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:10:48.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:10:48.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:10:48.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:10:48.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:10:48.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:10:48.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:10:48.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: ceph pg 2.0 query 2024-04-25T09:10:48.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:10:48.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: COUNT=22 2024-04-25T09:10:48.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: expr 11 '*' 2 2024-04-25T09:10:48.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: test 22 = 22 2024-04-25T09:10:48.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:234: TEST_rados_repair_warning: flush_pg_stats 2024-04-25T09:10:48.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:10:48.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:10:49.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:10:49.099 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:10:49.100 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:10:49.100 INFO:tasks.workunit.client.0.smithi102.stderr:3' 2024-04-25T09:10:49.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:10:49.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:49.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:10:49.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803831 2024-04-25T09:10:49.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803831 2024-04-25T09:10:49.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803831' 2024-04-25T09:10:49.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:49.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:10:49.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574892 2024-04-25T09:10:49.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574892 2024-04-25T09:10:49.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803831 1-55834574892' 2024-04-25T09:10:49.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:49.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:10:49.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313254 2024-04-25T09:10:49.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313254 2024-04-25T09:10:49.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803831 1-55834574892 2-90194313254' 2024-04-25T09:10:49.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:10:49.540 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:10:49.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018910 2024-04-25T09:10:49.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018910 2024-04-25T09:10:49.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803831 1-55834574892 2-90194313254 3-128849018910' 2024-04-25T09:10:49.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:49.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803831 2024-04-25T09:10:49.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:49.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:10:49.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803831 2024-04-25T09:10:49.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:49.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803831 2024-04-25T09:10:49.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803831' 2024-04-25T09:10:49.688 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803831 2024-04-25T09:10:49.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:10:50.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803829 -lt 25769803831 2024-04-25T09:10:50.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:10:51.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:10:51.141 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:10:51.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803831 -lt 25769803831 2024-04-25T09:10:51.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:51.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574892 2024-04-25T09:10:51.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:51.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:10:51.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574892 2024-04-25T09:10:51.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:51.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574892 2024-04-25T09:10:51.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574892' 2024-04-25T09:10:51.555 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574892 2024-04-25T09:10:51.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:10:51.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574892 -lt 55834574892 2024-04-25T09:10:51.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:51.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313254 2024-04-25T09:10:51.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:51.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:10:51.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313254 2024-04-25T09:10:51.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:51.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313254 2024-04-25T09:10:51.996 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 90194313254 2024-04-25T09:10:51.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313254' 2024-04-25T09:10:51.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:10:52.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313254 -lt 90194313254 2024-04-25T09:10:52.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:10:52.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:10:52.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-128849018910 2024-04-25T09:10:52.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:10:52.445 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-128849018910 2024-04-25T09:10:52.446 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:10:52.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018910 2024-04-25T09:10:52.447 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 128849018910 2024-04-25T09:10:52.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 128849018910' 2024-04-25T09:10:52.448 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:10:52.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018910 -lt 128849018910 2024-04-25T09:10:52.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-04-25T09:10:52.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:10:53.254 INFO:tasks.workunit.client.0.smithi102.stderr:dumped all 2024-04-25T09:10:53.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: COUNT=33 2024-04-25T09:10:53.271 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: expr 11 '*' 3 2024-04-25T09:10:53.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: test 33 = 33 2024-04-25T09:10:53.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:240: TEST_rados_repair_warning: CHECKTIME=10 2024-04-25T09:10:53.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:241: TEST_rados_repair_warning: LOOPS=0 2024-04-25T09:10:53.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:242: TEST_rados_repair_warning: true 2024-04-25T09:10:53.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:244: TEST_rados_repair_warning: sleep 1 2024-04-25T09:10:54.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: ceph health 2024-04-25T09:10:54.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 2 OSDs' 2024-04-25T09:10:54.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:247: TEST_rados_repair_warning: break 2024-04-25T09:10:54.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: ceph health detail 2024-04-25T09:10:54.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: expr 11 '*' 2 2024-04-25T09:10:54.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: grep -q 'osd.3 had 22 reads repaired' 2024-04-25T09:10:55.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: ceph health detail 2024-04-25T09:10:55.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: grep -q 'osd.1 had 11 reads repaired' 2024-04-25T09:10:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:259: TEST_rados_repair_warning: delete_pool pool-rep 2024-04-25T09:10:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-04-25T09:10:55.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-04-25T09:10:56.373 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'pool-rep' does not exist 2024-04-25T09:10:56.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:10:56.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:56.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:10:56.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:10:56.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:10:56.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:10:56.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:10:56.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:10:56.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:10:56.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:10:56.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:10:56.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:10:56.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:10:56.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:10:56.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:10:56.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:10:56.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:56.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:10:56.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:56.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:10:56.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:10:56.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:10:56.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:10:56.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:10:56.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:56.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:56.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.388535 2024-04-25T09:10:56.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:10:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:10:56.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:10:56.746 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:10:56.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:10:56.747 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:10:56.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:10:56.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:10:56.749 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:10:56.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:56.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:10:56.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:10:56.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:10:56.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:10:56.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:10:56.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:10:56.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:10:56.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:56.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:56.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.388535 2024-04-25T09:10:56.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:10:56.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:10:56.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T09:10:56.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:10:56.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:56.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:56.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.388535 2024-04-25T09:10:56.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:10:56.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:10:56.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:10:56.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T09:10:56.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T09:10:56.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:56.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:10:56.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:10:56.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:10:56.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T09:10:56.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T09:10:57.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:10:57.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:57.209 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:57.210 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:57.210 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:57.210 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:57.210 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:10:57.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T09:10:57.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:10:57.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:10:57.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:10:57.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:10:57.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:10:57.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:10:57.441 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:10:57.441 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:10:57.441 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:10:57.442 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:57.442 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:57.442 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:57.442 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-mon.a.asok 2024-04-25T09:10:57.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:10:57.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.388535/ceph-mon.a.asok config get fsid 2024-04-25T09:10:57.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:10:57.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:10:57.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:10:57.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:10:57.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:10:57.561 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:10:57.561 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:10:57.561 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:10:57.561 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:10:57.561 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:57.561 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:57.561 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-mon.a.asok 2024-04-25T09:10:57.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:10:57.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.388535/ceph-mon.a.asok config get mon_host 2024-04-25T09:10:57.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T09:10:57.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:57.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:10:57.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:10:57.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:10:57.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T09:10:57.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:10:58.574 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:10:58.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:58.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:58.575 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:58.575 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:58.575 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:58.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:10:58.576 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:10:58.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:10:58.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T09:10:58.940 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T09:10:58.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_backfill_unfound td/osd-rep-recov-eio.sh 2024-04-25T09:10:58.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:264: TEST_rep_backfill_unfound: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:58.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:265: TEST_rep_backfill_unfound: local objname=myobject 2024-04-25T09:10:58.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:266: TEST_rep_backfill_unfound: local lastobj=300 2024-04-25T09:10:58.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:268: TEST_rep_backfill_unfound: local testobj=obj250 2024-04-25T09:10:58.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:270: TEST_rep_backfill_unfound: export CEPH_ARGS 2024-04-25T09:10:58.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:271: TEST_rep_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:10:58.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:272: TEST_rep_backfill_unfound: setup_osds 3 2024-04-25T09:10:58.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-04-25T09:10:58.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T09:10:58.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T09:10:58.960 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-04-25T09:10:58.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-04-25T09:10:58.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:10:58.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:10:58.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:58.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:10:58.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:10:58.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:58.964 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:10:58.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:10:58.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:10:58.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:10:58.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:10:58.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:10:58.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:10:58.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:10:58.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:10:58.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:10:58.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:10:58.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:10:58.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:10:58.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:10:58.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:10:58.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:10:58.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=00c7da36-89ee-4dfd-983c-a83934988d53 2024-04-25T09:10:58.967 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 00c7da36-89ee-4dfd-983c-a83934988d53 2024-04-25T09:10:58.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 00c7da36-89ee-4dfd-983c-a83934988d53' 2024-04-25T09:10:58.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:10:58.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAjHipm7JhVABAA9YHuFy0YUMZDNE+OSiTuTQ== 2024-04-25T09:10:58.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAjHipm7JhVABAA9YHuFy0YUMZDNE+OSiTuTQ=="}' 2024-04-25T09:10:58.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 00c7da36-89ee-4dfd-983c-a83934988d53 -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:10:59.334 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:10:59.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:10:59.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAjHipm7JhVABAA9YHuFy0YUMZDNE+OSiTuTQ== --osd-uuid 00c7da36-89ee-4dfd-983c-a83934988d53 2024-04-25T09:10:59.386 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:10:59.399+0000 7f56c635e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:59.402 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:10:59.415+0000 7f56c635e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:59.403 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:10:59.415+0000 7f56c635e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:59.403 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:10:59.415+0000 7f56c635e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:10:59.404 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:10:59.415+0000 7f56c635e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:10:59.404 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:10:59.415+0000 7f56c635e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:10:59.442 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:10:59.455+0000 7f56c635e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T09:11:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T09:11:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:11:05.763 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:11:05.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:11:05.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:11:06.296 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:11:06.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:11:06.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:11:06.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:11:06.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:11:06.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:11:06.330 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:06.343+0000 7f3c299657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:06.332 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:06.343+0000 7f3c299657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:06.333 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:06.347+0000 7f3c299657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:06.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:11:06.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:11:06.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:11:06.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:11:06.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:11:06.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:06.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:11:06.739 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:11:06.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:06.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:06.886 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:06.899+0000 7f3c299657c0 -1 Falling back to public interface 2024-04-25T09:11:07.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:08.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:08.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:08.185 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:11:08.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:11:08.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:08.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:08.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:09.534 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:09.547+0000 7f3c299657c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:11:09.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:09.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:09.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:11:09.594 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:11:09.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:09.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:10.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:11.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:11.272 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:11:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:11:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:11.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:11.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:12.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:12.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:12.715 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:11:12.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:11:12.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:12.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:13.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:14.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:11:14.129 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:11:14.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:14.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:14.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:15.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:15.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:15.566 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:11:15.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:11:15.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:15.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:16.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:17.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:17.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:17.023 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:11:17.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:11:17.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:17.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:17.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:18.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:18.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:18.592 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:11:18.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:11:18.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:18.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:19.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:20.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:20.031 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:11:20.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:11:20.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:20.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:20.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:21.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:21.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:21.615 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:11:21.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:11:21.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:21.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:22.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:23.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:23.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:23.087 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:11:23.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:11:23.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:23.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:23.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:24.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:24.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:24.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:11:24.507 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:11:24.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:24.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:25.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:26.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:26.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:26.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:11:26.357 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:11:26.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:26.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:27.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:28.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:28.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:28.244 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:11:28.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:11:28.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:28.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:29.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:30.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:30.349 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:11:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:11:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:30.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:31.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:32.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:32.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:32.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:11:32.048 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:11:32.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:32.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:33.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:34.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:34.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:34.172 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:11:34.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:11:34.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:34.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:34.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:35.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:35.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:35.800 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:11:35.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:11:35.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:35.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:11:36.323 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2650463457,v1:127.0.0.1:6803/2650463457] [v2:127.0.0.1:6804/2650463457,v1:127.0.0.1:6805/2650463457] exists,up 00c7da36-89ee-4dfd-983c-a83934988d53 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:11:36.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:11:36.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:11:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:11:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:11:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:11:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:11:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:11:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:11:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:11:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:11:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:11:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:11:36.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T09:11:36.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:11:36.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6628ec72-85fe-43a3-8c99-166dbb80788a 2024-04-25T09:11:36.329 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6628ec72-85fe-43a3-8c99-166dbb80788a' 2024-04-25T09:11:36.329 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 6628ec72-85fe-43a3-8c99-166dbb80788a 2024-04-25T09:11:36.329 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:11:36.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBIHipm/gXPFRAAwPQ2YtKVt8HwBt/Gl/jS5g== 2024-04-25T09:11:36.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBIHipm/gXPFRAAwPQ2YtKVt8HwBt/Gl/jS5g=="}' 2024-04-25T09:11:36.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6628ec72-85fe-43a3-8c99-166dbb80788a -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:11:36.951 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:11:36.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:11:36.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBIHipm/gXPFRAAwPQ2YtKVt8HwBt/Gl/jS5g== --osd-uuid 6628ec72-85fe-43a3-8c99-166dbb80788a 2024-04-25T09:11:37.000 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:37.010+0000 7fb3aa4477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:37.011 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:37.022+0000 7fb3aa4477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:37.012 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:37.022+0000 7fb3aa4477c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:37.012 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:37.022+0000 7fb3aa4477c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:11:37.012 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:37.022+0000 7fb3aa4477c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:11:37.012 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:37.022+0000 7fb3aa4477c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:11:37.050 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:37.062+0000 7fb3aa4477c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T09:11:43.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T09:11:43.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:11:43.801 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:11:43.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:11:43.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:11:44.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:11:44.353 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:11:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:11:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:11:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:11:44.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:11:44.386 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:44.398+0000 7f8abe6977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:44.388 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:44.398+0000 7f8abe6977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:44.389 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:44.402+0000 7f8abe6977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:44.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:11:44.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:11:44.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:11:44.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:11:44.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:11:44.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:44.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:11:44.793 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:11:44.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:44.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:44.966 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:44.978+0000 7f8abe6977c0 -1 Falling back to public interface 2024-04-25T09:11:45.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:46.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:46.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:46.218 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:11:46.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:11:46.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:46.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:46.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:47.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:47.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:47.625 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:11:47.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:11:47.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:47.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:47.650 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:11:47.662+0000 7f8abe6977c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:11:48.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:49.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:49.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:49.051 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:11:49.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:11:49.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:49.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:49.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:50.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:50.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:50.819 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:11:50.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:11:50.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:50.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:51.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:52.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:52.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:52.104 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:11:52.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:11:52.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:52.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:52.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:53.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:53.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:53.608 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:11:53.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:11:53.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:53.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:54.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:55.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:55.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:55.235 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:11:55.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:11:55.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:55.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:55.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:56.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:56.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:56.773 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:11:56.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:11:56.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:56.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:57.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:58.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:58.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:58.227 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:11:58.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:11:58.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:58.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:11:58.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:59.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:59.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:59.723 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:11:59.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:11:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:59.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:00.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:01.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:01.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:01.327 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:12:01.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:12:01.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:01.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:01.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:02.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:02.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:02.768 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:12:02.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:12:02.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:02.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:03.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:04.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:04.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:04.347 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:12:04.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:12:04.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:04.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:05.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:06.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:06.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:06.088 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:12:06.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:12:06.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:06.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:07.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:08.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:08.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:08.052 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:12:08.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:12:08.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:08.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:08.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:09.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:09.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:09.840 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:12:09.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:12:09.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:09.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:11.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:12.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:12.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:12.122 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:12:12.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:12:12.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:12.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:13.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:14.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:14.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:14.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:12:14.361 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:12:14.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:14.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:12:15.000 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2815343810,v1:127.0.0.1:6811/2815343810] [v2:127.0.0.1:6812/2815343810,v1:127.0.0.1:6813/2815343810] exists,up 6628ec72-85fe-43a3-8c99-166dbb80788a 2024-04-25T09:12:15.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:12:15.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:12:15.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:12:15.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:12:15.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:12:15.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:12:15.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:12:15.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:12:15.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:12:15.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:12:15.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:12:15.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:12:15.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:12:15.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:12:15.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8eab7b2c-3450-4092-b021-1fc5ba05569b 2024-04-25T09:12:15.006 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 8eab7b2c-3450-4092-b021-1fc5ba05569b 2024-04-25T09:12:15.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8eab7b2c-3450-4092-b021-1fc5ba05569b' 2024-04-25T09:12:15.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:12:15.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvHipmjUFaAhAAcsxNzSrF1u87X01ZQ+Q/cg== 2024-04-25T09:12:15.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvHipmjUFaAhAAcsxNzSrF1u87X01ZQ+Q/cg=="}' 2024-04-25T09:12:15.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8eab7b2c-3450-4092-b021-1fc5ba05569b -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:12:15.896 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:12:15.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:12:15.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvHipmjUFaAhAAcsxNzSrF1u87X01ZQ+Q/cg== --osd-uuid 8eab7b2c-3450-4092-b021-1fc5ba05569b 2024-04-25T09:12:15.950 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:15.962+0000 7f29e41847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:15.979 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:15.990+0000 7f29e41847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:15.980 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:15.990+0000 7f29e41847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:15.980 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:15.994+0000 7f29e41847c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:12:15.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:15.994+0000 7f29e41847c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:12:15.981 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:15.994+0000 7f29e41847c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:12:16.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:16.030+0000 7f29e41847c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:12:23.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:12:23.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:12:23.047 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T09:12:23.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:12:23.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:12:23.665 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:12:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:12:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:12:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:12:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:12:23.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:12:23.699 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:23.710+0000 7f6316e267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:23.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:23.754+0000 7f6316e267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:23.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:23.754+0000 7f6316e267c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:12:24.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:12:24.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:12:24.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:12:24.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:12:24.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:12:24.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:24.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:12:24.142 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:12:24.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:24.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:24.331 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:24.341+0000 7f6316e267c0 -1 Falling back to public interface 2024-04-25T09:12:24.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:25.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:25.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:25.561 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:12:25.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:12:25.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:25.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:26.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:26.958 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:12:26.969+0000 7f6316e267c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:12:27.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:27.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:27.015 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:12:27.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:12:27.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:27.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:27.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:28.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:28.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:28.790 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:12:28.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:12:28.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:28.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:29.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:30.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:30.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:30.243 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:12:30.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:12:30.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:30.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:30.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:31.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:31.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:31.693 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:12:31.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:12:31.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:31.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:32.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:33.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:33.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:12:33.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:12:33.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:33.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:34.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:35.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:35.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:35.040 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:12:35.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:12:35.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:35.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:35.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:36.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:36.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:36.518 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:12:36.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:12:36.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:36.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:36.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:37.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:37.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:37.955 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:12:37.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:12:37.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:37.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:38.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:39.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:39.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:39.403 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:12:39.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:12:39.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:39.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:39.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:40.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:40.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:40.913 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:12:40.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:12:40.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:40.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:41.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:42.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:42.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:42.456 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:12:42.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:12:42.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:42.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:43.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:44.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:44.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:44.698 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:12:44.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:12:44.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:44.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:45.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:46.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:46.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:46.452 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:12:46.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:12:46.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:46.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:47.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:48.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:48.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:48.242 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:12:48.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:12:48.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:48.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:49.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:50.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:50.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:50.032 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:12:50.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:12:50.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:50.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:50.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:51.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:51.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:51.677 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:12:51.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:12:51.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:51.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:52.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:53.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:53.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:53.830 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:12:53.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:12:53.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:53.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:12:54.308 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4038367012,v1:127.0.0.1:6819/4038367012] [v2:127.0.0.1:6820/4038367012,v1:127.0.0.1:6821/4038367012] exists,up 8eab7b2c-3450-4092-b021-1fc5ba05569b 2024-04-25T09:12:54.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:12:54.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:12:54.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:12:54.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:12:54.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:12:54.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:12:54.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:12:54.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:12:54.310 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:12:54.310 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:12:54.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:12:54.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:12:54.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:12:54.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:12:54.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:12:54.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:12:54.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:12:54.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:12:54.413 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:12:54.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:12:54.807 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:12:54.807 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:12:54.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:12:54.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:54.807 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:12:54.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803800 2024-04-25T09:12:54.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803800 2024-04-25T09:12:54.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803800' 2024-04-25T09:12:54.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:54.988 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:12:55.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574864 2024-04-25T09:12:55.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574864 2024-04-25T09:12:55.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803800 1-55834574864' 2024-04-25T09:12:55.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:55.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:12:55.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345931 2024-04-25T09:12:55.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345931 2024-04-25T09:12:55.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803800 1-55834574864 2-85899345931' 2024-04-25T09:12:55.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:55.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803800 2024-04-25T09:12:55.422 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:55.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:12:55.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803800 2024-04-25T09:12:55.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:55.426 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803800 2024-04-25T09:12:55.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803800 2024-04-25T09:12:55.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803800' 2024-04-25T09:12:55.427 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:56.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803800 -lt 25769803800 2024-04-25T09:12:56.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:56.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574864 2024-04-25T09:12:56.349 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:56.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:12:56.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574864 2024-04-25T09:12:56.351 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:56.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574864 2024-04-25T09:12:56.353 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574864 2024-04-25T09:12:56.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574864' 2024-04-25T09:12:56.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:12:56.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574862 -lt 55834574864 2024-04-25T09:12:56.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:12:57.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:12:57.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:12:58.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574865 -lt 55834574864 2024-04-25T09:12:58.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:58.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:58.314 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345931 2024-04-25T09:12:58.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:12:58.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345931 2024-04-25T09:12:58.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345931 2024-04-25T09:12:58.317 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345931 2024-04-25T09:12:58.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345931' 2024-04-25T09:12:58.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:12:58.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345931 -lt 85899345931 2024-04-25T09:12:58.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:12:58.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:58.725 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:59.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:12:59.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:12:59.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:12:59.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:12:59.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:12:59.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:12:59.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:12:59.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:12:59.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:12:59.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:12:59.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:59.734 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:00.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 8 2024-04-25T09:13:00.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T09:13:00.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:13:00.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T09:13:00.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:00.309 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:13:00.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:13:00.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:00.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:00.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:00.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:00.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:00.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:00.412 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:00.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:13:00.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:00.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:00.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:01.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 8 2024-04-25T09:13:01.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:13:01.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:13:01.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:13:01.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:13:01.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:13:01.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:13:01.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:13:01.369 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:13:02.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:13:02.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:13:02.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T09:13:02.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:02.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T09:13:02.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:13:02.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:02.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:02.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:02.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:02.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:02.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:02.205 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:02.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:13:02.602 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:02.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:02.603 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:03.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:13:03.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:13:03.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:13:03.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:274: TEST_rep_backfill_unfound: local poolname=test-pool 2024-04-25T09:13:03.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:275: TEST_rep_backfill_unfound: create_pool test-pool 1 1 2024-04-25T09:13:03.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-04-25T09:13:03.738 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test-pool' already exists 2024-04-25T09:13:03.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:13:04.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:276: TEST_rep_backfill_unfound: wait_for_clean 2024-04-25T09:13:04.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:13:04.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:13:04.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:13:04.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:13:04.757 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:13:04.757 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:13:04.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:13:04.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:13:04.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:13:04.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:13:04.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:13:04.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:13:04.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:13:04.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:13:04.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:13:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:13:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:13:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:13:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:13:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:13:05.347 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:13:05.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803804 2024-04-25T09:13:05.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803804 2024-04-25T09:13:05.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804' 2024-04-25T09:13:05.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:13:05.509 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:13:05.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574869 2024-04-25T09:13:05.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574869 2024-04-25T09:13:05.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804 1-55834574869' 2024-04-25T09:13:05.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:13:05.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:13:05.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345935 2024-04-25T09:13:05.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345935 2024-04-25T09:13:05.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803804 1-55834574869 2-85899345935' 2024-04-25T09:13:05.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:13:05.820 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803804 2024-04-25T09:13:05.820 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:13:05.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:13:05.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803804 2024-04-25T09:13:05.822 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:13:05.823 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803804 2024-04-25T09:13:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803804 2024-04-25T09:13:05.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803804' 2024-04-25T09:13:05.823 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:13:06.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803802 -lt 25769803804 2024-04-25T09:13:06.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:13:07.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:13:07.243 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:13:07.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803805 -lt 25769803804 2024-04-25T09:13:07.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:13:07.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574869 2024-04-25T09:13:07.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:13:07.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:13:07.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574869 2024-04-25T09:13:07.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:13:07.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574869 2024-04-25T09:13:07.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574869' 2024-04-25T09:13:07.660 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574869 2024-04-25T09:13:07.660 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:13:08.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574869 -lt 55834574869 2024-04-25T09:13:08.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:13:08.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:13:08.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345935 2024-04-25T09:13:08.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:13:08.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345935 2024-04-25T09:13:08.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:13:08.062 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345935 2024-04-25T09:13:08.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345935 2024-04-25T09:13:08.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345935' 2024-04-25T09:13:08.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:13:08.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345935 -lt 85899345935 2024-04-25T09:13:08.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:13:08.483 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:08.484 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:09.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:13:09.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:09.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:09.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:09.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:09.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:09.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:09.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:09.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:13:09.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:09.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:09.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:10.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:13:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:13:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:13:10.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:278: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T09:13:10.395 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:13:10.395 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:56.444605+0000 0'0 26:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:16:08.725206+0000 0 0 2024-04-25T09:13:10.395 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:56.446188+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:37:52.159778+0000 0 0 2024-04-25T09:13:10.395 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:57.164959+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:55:54.251742+0000 0 0 2024-04-25T09:13:10.396 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:56.444200+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:06:05.038728+0000 0 0 2024-04-25T09:13:10.396 INFO:tasks.workunit.client.0.smithi102.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:13:04.734928+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:13:03.567791+0000 0'0 2024-04-25T09:13:03.567791+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:07:39.166053+0000 0 0 2024-04-25T09:13:10.396 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:56.445030+0000 0'0 26:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:13:38.794138+0000 0 0 2024-04-25T09:13:10.396 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:57.165345+0000 0'0 26:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:30:08.545418+0000 0 0 2024-04-25T09:13:10.396 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:57.030603+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:36:55.644508+0000 0 0 2024-04-25T09:13:10.396 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:56.444834+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:56:50.456134+0000 0 0 2024-04-25T09:13:10.396 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T09:13:10.396 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:13:10.396 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T09:13:10.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:280: TEST_rep_backfill_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-04-25T09:13:10.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:13:10.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-04-25T09:13:10.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-04-25T09:13:10.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:13:10.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:13:10.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:13:10.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:13:10.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:13:10.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:13:10.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:13:10.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:13:10.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:10.594 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: get_osds test-pool myobject 2024-04-25T09:13:10.594 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:13:10.594 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-25T09:13:10.595 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:13:10.595 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-04-25T09:13:11.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:13:11.033 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:13:11.033 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:13:11.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-25T09:13:11.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: initial_osds=('2' '1' '0') 2024-04-25T09:13:11.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: local -a initial_osds 2024-04-25T09:13:11.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:283: TEST_rep_backfill_unfound: local last_osd=0 2024-04-25T09:13:11.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:284: TEST_rep_backfill_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-04-25T09:13:11.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:13:11.033 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:13:11.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:13:11.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:13:11.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:13:11.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:13:11.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:285: TEST_rep_backfill_unfound: ceph osd down 0 2024-04-25T09:13:12.088 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T09:13:12.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:286: TEST_rep_backfill_unfound: ceph osd out 0 2024-04-25T09:13:12.845 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. 2024-04-25T09:13:12.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:288: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T09:13:13.270 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:13:13.270 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:56.444605+0000 0'0 26:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:16:08.725206+0000 0 0 2024-04-25T09:13:13.270 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:56.446188+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:37:52.159778+0000 0 0 2024-04-25T09:13:13.270 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:57.164959+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:55:54.251742+0000 0 0 2024-04-25T09:13:13.270 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:56.444200+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:06:05.038728+0000 0 0 2024-04-25T09:13:13.270 INFO:tasks.workunit.client.0.smithi102.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:13:04.734928+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:13:03.567791+0000 0'0 2024-04-25T09:13:03.567791+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:07:39.166053+0000 0 0 2024-04-25T09:13:13.270 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:56.445030+0000 0'0 26:35 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:13:38.794138+0000 0 0 2024-04-25T09:13:13.271 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2024-04-25T09:12:57.165345+0000 0'0 26:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:30:08.545418+0000 0 0 2024-04-25T09:13:13.271 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:57.030603+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:36:55.644508+0000 0 0 2024-04-25T09:13:13.271 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:12:56.444834+0000 0'0 26:35 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:56:50.456134+0000 0 0 2024-04-25T09:13:13.271 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T09:13:13.271 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:13:13.271 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T09:13:13.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:290: TEST_rep_backfill_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-04-25T09:13:13.287 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records in 2024-04-25T09:13:13.287 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records out 2024-04-25T09:13:13.287 INFO:tasks.workunit.client.0.smithi102.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000304215 s, 13.5 MB/s 2024-04-25T09:13:13.288 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: seq 1 300 2024-04-25T09:13:13.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:13.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:13.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:13.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:13.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:13.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:13.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:13.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:13.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:13.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:14.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:14.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:14.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:14.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:14.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:14.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:14.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:14.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:14.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:14.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:15.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:15.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:15.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:15.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:15.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:15.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:15.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:15.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:15.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:15.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:15.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:16.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:16.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:16.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:16.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:16.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:16.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:16.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:16.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:16.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:16.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:16.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:16.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:17.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:17.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:17.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:17.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:17.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:17.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:17.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:17.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:17.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:17.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:17.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:17.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:18.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:18.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:18.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:18.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:18.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:18.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:18.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:18.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:18.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:18.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:18.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:18.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:19.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:19.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:19.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:19.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:19.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:19.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:19.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:19.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:19.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:19.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:19.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:19.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:19.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:19.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:20.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:20.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:20.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:20.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:20.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:20.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:20.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:20.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:20.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:20.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:20.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:20.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:21.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:21.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:21.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:21.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:21.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:21.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:21.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:21.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:21.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:21.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:21.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:21.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:22.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:22.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:22.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:22.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:22.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:22.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:22.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:22.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:22.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:22.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:22.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:22.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:23.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:23.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:23.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:23.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:23.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:23.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:23.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:23.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:23.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:23.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:24.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:24.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:24.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:24.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:24.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:24.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:24.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:24.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:24.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:24.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:24.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:24.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:25.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:25.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:25.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:25.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:25.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:25.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:25.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:25.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:25.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:25.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:25.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:25.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:26.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:26.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:26.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:26.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:26.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:26.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:26.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:26.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:26.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:26.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:26.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:26.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:27.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:27.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:27.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:27.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:27.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:27.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:27.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:27.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:27.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:27.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:27.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:27.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:28.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:28.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:28.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:28.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:28.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:28.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:28.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:28.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:28.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:28.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:28.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:29.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:29.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:29.206 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:29.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:29.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:29.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:29.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:29.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:29.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:29.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj101 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:29.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:29.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj102 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:30.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:30.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj103 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:30.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:30.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj104 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:30.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:30.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj105 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:30.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:30.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj106 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:30.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:30.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj107 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:30.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:30.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj108 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:31.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:31.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj109 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:31.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:31.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj110 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:31.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:31.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj111 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:31.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:31.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj112 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:31.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:31.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj113 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:32.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:32.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj114 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:32.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:32.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj115 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:32.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:32.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj116 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:32.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:32.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj117 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:32.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:32.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj118 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:32.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:32.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj119 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:33.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:33.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj120 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:33.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:33.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj121 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:33.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:33.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj122 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:33.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:33.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj123 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:33.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj124 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:33.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:33.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj125 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:34.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:34.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj126 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:34.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj127 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:34.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:34.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj128 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:34.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:34.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj129 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:34.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:34.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj130 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:34.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:34.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj131 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:35.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:35.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj132 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:35.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:35.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj133 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:35.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:35.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj134 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:35.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj135 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:35.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:35.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj136 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:35.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:35.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj137 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:35.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:35.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj138 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:36.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:36.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj139 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:36.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:36.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj140 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:36.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:36.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj141 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:36.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:36.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj142 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:36.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:36.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj143 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:36.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:36.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj144 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:37.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:37.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj145 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:37.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:37.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj146 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:37.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:37.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj147 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:37.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj148 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:37.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:37.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj149 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:37.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:37.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj150 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:38.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:38.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj151 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:38.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:38.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj152 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:38.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:38.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj153 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:38.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:38.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj154 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:38.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:38.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj155 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:38.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:38.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj156 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:39.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:39.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj157 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:39.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:39.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj158 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:39.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:39.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj159 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:39.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:39.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj160 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:39.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:39.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj161 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:39.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:39.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj162 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:40.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:40.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj163 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:40.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:40.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj164 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:40.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:40.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj165 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:40.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:40.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj166 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:40.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:40.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj167 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:40.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:40.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj168 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:41.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:41.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj169 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:41.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:41.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj170 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:41.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:41.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj171 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:41.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:41.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj172 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:41.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:41.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj173 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:41.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:41.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj174 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:42.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:42.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj175 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:42.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:42.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj176 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:42.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj177 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:42.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:42.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj178 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:42.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:42.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj179 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:42.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:42.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj180 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:43.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:43.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj181 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:43.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:43.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj182 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:43.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:43.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj183 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:43.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:43.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj184 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:43.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:43.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj185 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:43.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:43.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj186 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:44.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:44.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj187 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:44.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:44.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj188 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:44.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:44.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj189 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:44.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:44.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj190 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:44.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:44.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj191 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:44.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:44.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj192 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:44.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:44.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj193 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:45.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:45.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj194 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:45.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:45.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj195 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:45.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:45.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj196 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:45.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:45.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj197 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:45.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:45.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj198 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:45.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:45.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj199 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:46.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:46.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj200 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:46.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:46.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj201 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:46.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:46.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj202 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:46.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:46.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj203 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:46.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:46.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj204 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:46.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:46.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj205 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:47.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:47.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj206 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:47.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:47.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj207 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:47.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:47.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj208 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:47.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:47.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj209 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:47.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:47.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj210 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:47.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:47.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj211 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:48.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:48.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj212 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:48.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:48.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj213 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:48.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:48.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj214 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:48.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:48.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj215 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:48.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:48.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj216 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:48.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:48.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj217 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:49.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:49.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj218 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:49.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:49.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj219 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:49.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:49.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj220 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:49.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:49.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj221 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:49.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:49.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj222 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:49.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:49.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj223 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:50.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:50.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj224 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:50.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:50.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj225 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:50.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:50.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj226 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:50.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:50.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj227 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:50.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:50.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj228 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:50.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:50.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj229 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:51.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:51.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj230 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:51.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:51.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj231 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:51.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:51.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj232 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:51.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:51.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj233 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:51.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:51.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj234 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:52.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:52.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj235 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:52.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:52.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj236 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:52.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:52.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj237 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:52.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:52.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj238 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:52.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:52.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj239 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:52.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:52.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj240 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:52.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:52.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj241 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:53.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:53.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj242 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:53.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:53.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj243 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:53.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:53.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj244 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:53.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:53.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj245 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:53.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:53.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj246 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:53.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:53.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj247 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:54.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:54.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj248 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:54.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:54.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj249 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:54.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:54.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj250 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:54.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj251 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:54.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:54.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj252 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:54.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:54.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj253 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:55.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:55.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj254 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:55.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:55.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj255 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:55.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:55.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj256 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:55.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:55.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj257 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:55.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:55.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj258 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:55.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:55.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj259 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:56.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:56.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj260 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj261 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:56.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:56.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj262 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:56.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:56.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj263 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:56.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:56.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj264 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:56.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:56.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj265 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:57.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:57.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj266 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:57.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:57.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj267 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:57.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:57.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj268 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:57.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:57.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj269 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:57.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:57.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj270 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:57.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:57.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj271 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:58.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:58.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj272 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:58.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:58.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj273 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:58.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:58.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj274 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:58.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:58.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj275 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:58.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:58.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj276 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:58.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:58.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj277 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:59.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:59.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj278 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:59.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:59.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj279 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:59.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:59.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj280 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:59.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:59.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj281 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:59.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:59.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj282 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:13:59.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:13:59.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj283 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:00.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:00.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj284 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:00.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:00.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj285 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:00.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:00.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj286 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:00.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:00.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj287 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:00.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:00.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj288 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:00.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:00.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj289 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:01.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:01.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj290 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:01.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:01.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj291 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:01.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:01.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj292 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:01.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:01.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj293 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:01.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:01.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj294 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:02.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:02.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj295 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:02.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:02.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj296 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:02.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:02.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj297 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:02.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:02.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj298 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:02.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:02.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj299 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:03.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:03.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj300 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:14:03.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:296: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 0 2024-04-25T09:14:03.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:14:03.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:14:03.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:14:03.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:14:03.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:14:03.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:14:03.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj250 2024-04-25T09:14:03.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:14:03.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:14:03.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:14:03.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:14:03.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:14:03.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj250 2024-04-25T09:14:03.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:14:03.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-04-25T09:14:03.251 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-04-25T09:14:03.251 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:14:03.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:14:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-04-25T09:14:03.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:14:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:14:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:14:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=2 2024-04-25T09:14:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:14:03.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:14:03.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-04-25T09:14:03.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:14:03.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-04-25T09:14:03.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:14:03.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-04-25T09:14:03.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:14:03.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:14:03.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:14:03.690 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-04-25T09:14:03.690 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:14:03.690 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:14:03.690 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:14:03.690 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:03.690 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:14:03.690 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.2.asok 2024-04-25T09:14:03.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:14:03.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:14:03.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:14:03.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:14:03.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.2 2024-04-25T09:14:03.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:14:03.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:14:03.809 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:14:03.809 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:03.809 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:14:03.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.2.asok 2024-04-25T09:14:03.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:14:03.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.2.asok injectdataerr test-pool obj250 2024-04-25T09:14:03.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:297: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 1 2024-04-25T09:14:03.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:14:03.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:14:03.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:14:03.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:14:03.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:14:03.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:14:03.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj250 2024-04-25T09:14:03.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:14:03.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:14:03.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:14:03.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:14:03.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:14:03.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj250 2024-04-25T09:14:03.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:14:03.920 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-04-25T09:14:03.920 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:14:03.920 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-04-25T09:14:04.339 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:14:04.340 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-04-25T09:14:04.340 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:14:04.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:14:04.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:14:04.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:14:04.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:14:04.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:14:04.340 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:14:04.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:14:04.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:14:04.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:14:04.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:14:04.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:14:04.341 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:14:04.342 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:14:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:14:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:14:04.342 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:14:04.343 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:14:04.343 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:04.343 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:14:04.343 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:14:04.343 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:14:04.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:14:04.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:14:04.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:14:04.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:14:04.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:14:04.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:14:04.452 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:14:04.453 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:04.453 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:14:04.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:14:04.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:14:04.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr test-pool obj250 2024-04-25T09:14:04.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:299: TEST_rep_backfill_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:14:04.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:14:04.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:14:04.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:14:04.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:14:04.564 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:14:04.564 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:14:04.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:14:04.566 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:14:04.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:14:04.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:14:04.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-04-25T09:14:04.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:14:04.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:14:04.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:14:04.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:14:04.598 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:14:04.612+0000 7feed9a6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:04.600 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:14:04.616+0000 7feed9a6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:04.601 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:14:04.616+0000 7feed9a6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:04.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:14:04.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:14:04.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:14:04.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:14:04.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:14:04.995 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:14:04.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:04.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:14:04.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:14:04.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:05.141 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:14:05.156+0000 7feed9a6b7c0 -1 Falling back to public interface 2024-04-25T09:14:05.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:06.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:06.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:06.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:14:06.458 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:14:06.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:06.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:14:06.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:07.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:07.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:07.864 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:14:07.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:14:07.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:07.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:14:08.271 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:14:08.283+0000 7feed9a6b7c0 -1 osd.0 26 log_to_monitors true 2024-04-25T09:14:08.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:14:09.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:14:09.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:14:09.282 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:14:09.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:14:09.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:14:09.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:14:10.116 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up out weight 0 up_from 32 up_thru 21 down_at 27 last_clean_interval [6,26) [v2:127.0.0.1:6802/1453865232,v1:127.0.0.1:6803/1453865232] [v2:127.0.0.1:6804/1453865232,v1:127.0.0.1:6805/1453865232] exists,up 00c7da36-89ee-4dfd-983c-a83934988d53 2024-04-25T09:14:10.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:14:10.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:14:10.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:14:10.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:300: TEST_rep_backfill_unfound: ceph osd in 0 2024-04-25T09:14:11.690 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already in. 2024-04-25T09:14:11.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:302: TEST_rep_backfill_unfound: sleep 15 2024-04-25T09:14:26.716 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: seq 1 360 2024-04-25T09:14:26.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:14:26.718 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:14:26.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:14:26.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:14:26.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:14:26.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:14:27.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:27.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:14:27.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:27.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:14:27.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:14:27.131 INFO:tasks.workunit.client.0.smithi102.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:27.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:14:28.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:14:28.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:14:28.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:14:28.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:14:28.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:14:28.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:14:28.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:28.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:28.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:14:28.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:14:28.547 INFO:tasks.workunit.client.0.smithi102.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:28.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:14:28.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:14:29.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:14:29.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:14:29.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:14:29.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:14:29.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:14:29.564 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:14:29.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:29.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:29.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:14:29.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:14:29.967 INFO:tasks.workunit.client.0.smithi102.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:14:29.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:14:30.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:14:30.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:14:30.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:14:30.970 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:14:30.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:14:30.971 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:14:31.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:31.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:31.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:14:31.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:14:31.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:14:31.401 INFO:tasks.workunit.client.0.smithi102.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:31.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:14:32.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:14:32.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:14:32.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:14:32.403 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:14:32.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:14:32.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:14:32.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:32.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:32.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:14:32.835 INFO:tasks.workunit.client.0.smithi102.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:32.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:14:32.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:14:32.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:14:33.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:14:33.838 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:14:33.838 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:14:33.838 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:14:33.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:14:33.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:14:34.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:34.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:34.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:14:34.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:14:34.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:14:34.265 INFO:tasks.workunit.client.0.smithi102.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:14:34.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:14:35.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:14:35.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:14:35.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:14:35.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:14:35.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:14:35.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:14:35.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:14:35.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:14:35.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:14:35.678 INFO:tasks.workunit.client.0.smithi102.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:14:35.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 0 = 0 ']' 2024-04-25T09:14:35.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:308: TEST_rep_backfill_unfound: break 2024-04-25T09:14:35.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:314: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T09:14:36.091 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:14:36.092 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:14:12.490992+0000 0'0 41:70 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:58:21.660885+0000 0 0 2024-04-25T09:14:36.092 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:14:13.492118+0000 0'0 41:68 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:06:32.080471+0000 0 0 2024-04-25T09:14:36.092 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:14:13.455307+0000 0'0 41:58 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:28:13.055553+0000 0 0 2024-04-25T09:14:36.092 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:14:13.453572+0000 0'0 41:68 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:20:07.733769+0000 0 0 2024-04-25T09:14:36.092 INFO:tasks.workunit.client.0.smithi102.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:14:34.514089+0000 30'301 41:895 [2,1,0] 2 [2,1] 2 0'0 2024-04-25T09:13:03.567791+0000 0'0 2024-04-25T09:13:03.567791+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:14:36.092 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:14:12.490624+0000 0'0 41:70 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:57:28.575486+0000 0 0 2024-04-25T09:14:36.092 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:14:12.934884+0000 0'0 41:56 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:24:27.527868+0000 0 0 2024-04-25T09:14:36.092 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:14:13.454557+0000 0'0 41:58 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:14:49.664274+0000 0 0 2024-04-25T09:14:36.092 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:14:13.493271+0000 0'0 41:68 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:10:58.807701+0000 0'0 2024-04-25T09:10:58.807701+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:15:36.890022+0000 0 0 2024-04-25T09:14:36.092 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T09:14:36.092 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:14:36.093 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T09:14:36.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: ceph pg 2.0 list_unfound 2024-04-25T09:14:36.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: grep -q obj250 2024-04-25T09:14:36.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:318: TEST_rep_backfill_unfound: timeout 5 rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:41.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:319: TEST_rep_backfill_unfound: test 124 = 124 2024-04-25T09:14:41.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:321: TEST_rep_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-04-25T09:14:41.513 INFO:tasks.workunit.client.0.smithi102.stderr:pg has no unfound objects 2024-04-25T09:14:41.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:323: TEST_rep_backfill_unfound: wait_for_clean 2024-04-25T09:14:41.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:14:41.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:14:41.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:14:41.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:14:41.533 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:14:41.534 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:14:41.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:14:41.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:14:41.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:14:41.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:14:41.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:14:41.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:14:41.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:14:41.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:14:41.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:14:42.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:14:42.071 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:14:42.071 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:14:42.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:14:42.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:42.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:14:42.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953481 2024-04-25T09:14:42.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953481 2024-04-25T09:14:42.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953481' 2024-04-25T09:14:42.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:42.232 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:14:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574890 2024-04-25T09:14:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574890 2024-04-25T09:14:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953481 1-55834574890' 2024-04-25T09:14:42.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:42.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:14:42.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345956 2024-04-25T09:14:42.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345956 2024-04-25T09:14:42.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953481 1-55834574890 2-85899345956' 2024-04-25T09:14:42.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:42.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-137438953481 2024-04-25T09:14:42.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:42.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:14:42.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-137438953481 2024-04-25T09:14:42.549 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:42.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953481 2024-04-25T09:14:42.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 137438953481' 2024-04-25T09:14:42.550 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 137438953481 2024-04-25T09:14:42.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:42.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953479 -lt 137438953481 2024-04-25T09:14:42.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:14:43.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:14:43.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:44.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953481 -lt 137438953481 2024-04-25T09:14:44.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:44.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574890 2024-04-25T09:14:44.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:44.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:14:44.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574890 2024-04-25T09:14:44.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:44.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574890 2024-04-25T09:14:44.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574890' 2024-04-25T09:14:44.419 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574890 2024-04-25T09:14:44.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:14:45.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574890 -lt 55834574890 2024-04-25T09:14:45.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:45.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:45.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345956 2024-04-25T09:14:45.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:14:45.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345956 2024-04-25T09:14:45.061 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:45.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345956 2024-04-25T09:14:45.063 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345956 2024-04-25T09:14:45.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345956' 2024-04-25T09:14:45.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:14:45.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345957 -lt 85899345956 2024-04-25T09:14:45.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:14:45.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:45.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:46.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:14:46.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:46.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:46.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:46.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:46.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:46.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:46.268 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:46.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:14:46.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:46.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:46.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:47.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:14:47.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-04-25T09:14:47.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:14:47.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-04-25T09:14:47.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:14:47.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:14:47.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:14:47.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:47.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:47.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:47.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:47.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:47.334 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:47.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:47.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:14:47.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:47.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:47.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:48.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:14:48.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:14:48.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:14:48.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:14:48.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:14:48.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:14:48.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:14:48.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:14:48.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:14:48.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12799 2024-04-25T09:14:48.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12799 '!=' null 2024-04-25T09:14:48.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:14:48.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:14:48.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:14:48.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:14:48.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:48.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:48.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:48.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:48.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:48.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:48.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:49.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:14:49.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:49.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:49.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:49.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:14:49.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:14:49.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:14:49.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:14:49.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:14:49.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:14:49.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:14:49.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:14:49.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:14:50.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12799 2024-04-25T09:14:50.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12799 '!=' null 2024-04-25T09:14:50.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:14:50.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:14:50.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:14:50.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:14:50.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:50.503 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:50.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:50.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:50.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:50.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:50.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:51.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:14:51.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:14:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:14:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:14:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T09:14:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:14:52.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T09:14:52.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:14:52.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:52.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:52.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:52.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:52.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:52.240 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:52.241 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:52.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:14:52.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:52.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:52.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:53.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:14:53.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:14:53.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:14:53.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: seq 1 300 2024-04-25T09:14:53.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj1 = obj250 ']' 2024-04-25T09:14:53.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj2 = obj250 ']' 2024-04-25T09:14:53.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj3 = obj250 ']' 2024-04-25T09:14:53.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj4 = obj250 ']' 2024-04-25T09:14:53.326 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj5 = obj250 ']' 2024-04-25T09:14:53.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj6 = obj250 ']' 2024-04-25T09:14:53.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj7 = obj250 ']' 2024-04-25T09:14:53.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj8 = obj250 ']' 2024-04-25T09:14:53.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj9 = obj250 ']' 2024-04-25T09:14:53.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj10 = obj250 ']' 2024-04-25T09:14:53.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj11 = obj250 ']' 2024-04-25T09:14:53.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj12 = obj250 ']' 2024-04-25T09:14:53.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj13 = obj250 ']' 2024-04-25T09:14:53.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.819 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj14 = obj250 ']' 2024-04-25T09:14:53.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj15 = obj250 ']' 2024-04-25T09:14:53.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj16 = obj250 ']' 2024-04-25T09:14:53.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj17 = obj250 ']' 2024-04-25T09:14:53.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj18 = obj250 ']' 2024-04-25T09:14:53.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:53.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:53.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj19 = obj250 ']' 2024-04-25T09:14:53.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj20 = obj250 ']' 2024-04-25T09:14:54.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj21 = obj250 ']' 2024-04-25T09:14:54.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj22 = obj250 ']' 2024-04-25T09:14:54.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj23 = obj250 ']' 2024-04-25T09:14:54.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj24 = obj250 ']' 2024-04-25T09:14:54.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj25 = obj250 ']' 2024-04-25T09:14:54.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj26 = obj250 ']' 2024-04-25T09:14:54.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj27 = obj250 ']' 2024-04-25T09:14:54.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj28 = obj250 ']' 2024-04-25T09:14:54.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj29 = obj250 ']' 2024-04-25T09:14:54.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj30 = obj250 ']' 2024-04-25T09:14:54.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj31 = obj250 ']' 2024-04-25T09:14:54.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj32 = obj250 ']' 2024-04-25T09:14:54.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj33 = obj250 ']' 2024-04-25T09:14:54.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj34 = obj250 ']' 2024-04-25T09:14:54.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj35 = obj250 ']' 2024-04-25T09:14:54.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj36 = obj250 ']' 2024-04-25T09:14:54.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj37 = obj250 ']' 2024-04-25T09:14:54.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj38 = obj250 ']' 2024-04-25T09:14:54.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj39 = obj250 ']' 2024-04-25T09:14:54.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj40 = obj250 ']' 2024-04-25T09:14:54.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj41 = obj250 ']' 2024-04-25T09:14:54.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj42 = obj250 ']' 2024-04-25T09:14:54.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj43 = obj250 ']' 2024-04-25T09:14:54.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj44 = obj250 ']' 2024-04-25T09:14:54.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj45 = obj250 ']' 2024-04-25T09:14:54.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj46 = obj250 ']' 2024-04-25T09:14:54.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj47 = obj250 ']' 2024-04-25T09:14:54.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj48 = obj250 ']' 2024-04-25T09:14:54.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj49 = obj250 ']' 2024-04-25T09:14:54.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj50 = obj250 ']' 2024-04-25T09:14:54.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj51 = obj250 ']' 2024-04-25T09:14:54.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj52 = obj250 ']' 2024-04-25T09:14:54.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj53 = obj250 ']' 2024-04-25T09:14:54.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj54 = obj250 ']' 2024-04-25T09:14:55.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj55 = obj250 ']' 2024-04-25T09:14:55.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj56 = obj250 ']' 2024-04-25T09:14:55.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj57 = obj250 ']' 2024-04-25T09:14:55.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj58 = obj250 ']' 2024-04-25T09:14:55.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj59 = obj250 ']' 2024-04-25T09:14:55.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj60 = obj250 ']' 2024-04-25T09:14:55.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj61 = obj250 ']' 2024-04-25T09:14:55.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj62 = obj250 ']' 2024-04-25T09:14:55.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj63 = obj250 ']' 2024-04-25T09:14:55.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj64 = obj250 ']' 2024-04-25T09:14:55.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj65 = obj250 ']' 2024-04-25T09:14:55.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj66 = obj250 ']' 2024-04-25T09:14:55.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj67 = obj250 ']' 2024-04-25T09:14:55.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj68 = obj250 ']' 2024-04-25T09:14:55.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj69 = obj250 ']' 2024-04-25T09:14:55.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj70 = obj250 ']' 2024-04-25T09:14:55.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj71 = obj250 ']' 2024-04-25T09:14:55.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj72 = obj250 ']' 2024-04-25T09:14:55.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj73 = obj250 ']' 2024-04-25T09:14:55.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj74 = obj250 ']' 2024-04-25T09:14:55.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj75 = obj250 ']' 2024-04-25T09:14:55.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj76 = obj250 ']' 2024-04-25T09:14:55.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj77 = obj250 ']' 2024-04-25T09:14:55.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj78 = obj250 ']' 2024-04-25T09:14:55.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj79 = obj250 ']' 2024-04-25T09:14:55.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj80 = obj250 ']' 2024-04-25T09:14:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj81 = obj250 ']' 2024-04-25T09:14:55.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj82 = obj250 ']' 2024-04-25T09:14:55.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:55.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj83 = obj250 ']' 2024-04-25T09:14:55.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:55.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj84 = obj250 ']' 2024-04-25T09:14:56.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj85 = obj250 ']' 2024-04-25T09:14:56.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj86 = obj250 ']' 2024-04-25T09:14:56.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj87 = obj250 ']' 2024-04-25T09:14:56.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj88 = obj250 ']' 2024-04-25T09:14:56.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj89 = obj250 ']' 2024-04-25T09:14:56.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj90 = obj250 ']' 2024-04-25T09:14:56.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj91 = obj250 ']' 2024-04-25T09:14:56.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj92 = obj250 ']' 2024-04-25T09:14:56.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj93 = obj250 ']' 2024-04-25T09:14:56.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj94 = obj250 ']' 2024-04-25T09:14:56.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj95 = obj250 ']' 2024-04-25T09:14:56.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj96 = obj250 ']' 2024-04-25T09:14:56.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj97 = obj250 ']' 2024-04-25T09:14:56.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj98 = obj250 ']' 2024-04-25T09:14:56.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj99 = obj250 ']' 2024-04-25T09:14:56.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj100 = obj250 ']' 2024-04-25T09:14:56.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj101 = obj250 ']' 2024-04-25T09:14:56.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj101 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj102 = obj250 ']' 2024-04-25T09:14:56.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj102 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj103 = obj250 ']' 2024-04-25T09:14:56.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj103 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj104 = obj250 ']' 2024-04-25T09:14:56.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj104 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.623 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj105 = obj250 ']' 2024-04-25T09:14:56.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj105 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj106 = obj250 ']' 2024-04-25T09:14:56.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj106 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj107 = obj250 ']' 2024-04-25T09:14:56.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj107 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj108 = obj250 ']' 2024-04-25T09:14:56.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj108 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj109 = obj250 ']' 2024-04-25T09:14:56.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj109 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj110 = obj250 ']' 2024-04-25T09:14:56.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj110 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj111 = obj250 ']' 2024-04-25T09:14:56.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj111 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj112 = obj250 ']' 2024-04-25T09:14:56.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj112 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj113 = obj250 ']' 2024-04-25T09:14:56.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj113 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj114 = obj250 ']' 2024-04-25T09:14:56.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj114 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj115 = obj250 ']' 2024-04-25T09:14:56.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj115 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj116 = obj250 ']' 2024-04-25T09:14:56.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj116 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:56.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:56.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj117 = obj250 ']' 2024-04-25T09:14:56.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj117 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj118 = obj250 ']' 2024-04-25T09:14:57.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj118 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj119 = obj250 ']' 2024-04-25T09:14:57.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj119 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj120 = obj250 ']' 2024-04-25T09:14:57.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj120 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj121 = obj250 ']' 2024-04-25T09:14:57.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj121 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj122 = obj250 ']' 2024-04-25T09:14:57.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj122 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj123 = obj250 ']' 2024-04-25T09:14:57.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj123 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj124 = obj250 ']' 2024-04-25T09:14:57.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj124 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj125 = obj250 ']' 2024-04-25T09:14:57.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj125 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj126 = obj250 ']' 2024-04-25T09:14:57.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj126 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj127 = obj250 ']' 2024-04-25T09:14:57.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj127 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj128 = obj250 ']' 2024-04-25T09:14:57.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj128 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj129 = obj250 ']' 2024-04-25T09:14:57.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj129 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj130 = obj250 ']' 2024-04-25T09:14:57.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj130 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj131 = obj250 ']' 2024-04-25T09:14:57.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj131 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj132 = obj250 ']' 2024-04-25T09:14:57.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj132 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj133 = obj250 ']' 2024-04-25T09:14:57.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj133 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj134 = obj250 ']' 2024-04-25T09:14:57.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj134 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj135 = obj250 ']' 2024-04-25T09:14:57.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj135 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj136 = obj250 ']' 2024-04-25T09:14:57.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj136 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj137 = obj250 ']' 2024-04-25T09:14:57.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj137 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj138 = obj250 ']' 2024-04-25T09:14:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj138 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj139 = obj250 ']' 2024-04-25T09:14:57.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj139 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj140 = obj250 ']' 2024-04-25T09:14:57.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj140 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj141 = obj250 ']' 2024-04-25T09:14:57.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj141 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj142 = obj250 ']' 2024-04-25T09:14:57.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj142 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj143 = obj250 ']' 2024-04-25T09:14:57.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj143 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj144 = obj250 ']' 2024-04-25T09:14:57.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj144 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj145 = obj250 ']' 2024-04-25T09:14:57.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj145 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj146 = obj250 ']' 2024-04-25T09:14:57.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj146 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj147 = obj250 ']' 2024-04-25T09:14:57.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj147 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:57.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:57.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj148 = obj250 ']' 2024-04-25T09:14:57.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj148 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj149 = obj250 ']' 2024-04-25T09:14:58.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj149 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj150 = obj250 ']' 2024-04-25T09:14:58.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj150 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj151 = obj250 ']' 2024-04-25T09:14:58.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj151 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj152 = obj250 ']' 2024-04-25T09:14:58.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj152 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj153 = obj250 ']' 2024-04-25T09:14:58.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj153 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj154 = obj250 ']' 2024-04-25T09:14:58.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj154 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj155 = obj250 ']' 2024-04-25T09:14:58.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj155 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj156 = obj250 ']' 2024-04-25T09:14:58.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj156 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj157 = obj250 ']' 2024-04-25T09:14:58.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj157 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj158 = obj250 ']' 2024-04-25T09:14:58.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj158 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj159 = obj250 ']' 2024-04-25T09:14:58.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj159 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj160 = obj250 ']' 2024-04-25T09:14:58.328 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj160 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj161 = obj250 ']' 2024-04-25T09:14:58.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj161 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj162 = obj250 ']' 2024-04-25T09:14:58.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj162 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj163 = obj250 ']' 2024-04-25T09:14:58.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj163 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj164 = obj250 ']' 2024-04-25T09:14:58.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj164 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj165 = obj250 ']' 2024-04-25T09:14:58.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj165 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj166 = obj250 ']' 2024-04-25T09:14:58.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj166 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj167 = obj250 ']' 2024-04-25T09:14:58.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj167 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj168 = obj250 ']' 2024-04-25T09:14:58.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj168 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj169 = obj250 ']' 2024-04-25T09:14:58.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj169 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj170 = obj250 ']' 2024-04-25T09:14:58.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj170 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj171 = obj250 ']' 2024-04-25T09:14:58.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj171 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj172 = obj250 ']' 2024-04-25T09:14:58.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj172 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj173 = obj250 ']' 2024-04-25T09:14:58.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj173 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj174 = obj250 ']' 2024-04-25T09:14:58.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj174 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj175 = obj250 ']' 2024-04-25T09:14:58.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj175 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj176 = obj250 ']' 2024-04-25T09:14:58.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj176 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj177 = obj250 ']' 2024-04-25T09:14:58.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj177 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj178 = obj250 ']' 2024-04-25T09:14:58.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj178 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj179 = obj250 ']' 2024-04-25T09:14:58.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj179 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj180 = obj250 ']' 2024-04-25T09:14:58.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj180 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj181 = obj250 ']' 2024-04-25T09:14:58.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj181 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:58.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:58.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj182 = obj250 ']' 2024-04-25T09:14:58.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj182 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj183 = obj250 ']' 2024-04-25T09:14:59.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj183 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj184 = obj250 ']' 2024-04-25T09:14:59.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj184 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj185 = obj250 ']' 2024-04-25T09:14:59.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj185 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj186 = obj250 ']' 2024-04-25T09:14:59.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj186 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj187 = obj250 ']' 2024-04-25T09:14:59.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj187 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj188 = obj250 ']' 2024-04-25T09:14:59.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj188 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj189 = obj250 ']' 2024-04-25T09:14:59.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj189 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj190 = obj250 ']' 2024-04-25T09:14:59.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj190 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj191 = obj250 ']' 2024-04-25T09:14:59.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj191 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj192 = obj250 ']' 2024-04-25T09:14:59.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj192 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj193 = obj250 ']' 2024-04-25T09:14:59.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj193 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj194 = obj250 ']' 2024-04-25T09:14:59.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj194 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj195 = obj250 ']' 2024-04-25T09:14:59.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj195 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj196 = obj250 ']' 2024-04-25T09:14:59.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj196 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj197 = obj250 ']' 2024-04-25T09:14:59.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj197 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj198 = obj250 ']' 2024-04-25T09:14:59.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj198 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj199 = obj250 ']' 2024-04-25T09:14:59.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj199 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj200 = obj250 ']' 2024-04-25T09:14:59.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj200 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj201 = obj250 ']' 2024-04-25T09:14:59.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj201 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj202 = obj250 ']' 2024-04-25T09:14:59.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj202 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj203 = obj250 ']' 2024-04-25T09:14:59.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj203 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj204 = obj250 ']' 2024-04-25T09:14:59.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj204 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj205 = obj250 ']' 2024-04-25T09:14:59.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj205 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj206 = obj250 ']' 2024-04-25T09:14:59.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj206 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj207 = obj250 ']' 2024-04-25T09:14:59.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj207 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj208 = obj250 ']' 2024-04-25T09:14:59.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj208 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj209 = obj250 ']' 2024-04-25T09:14:59.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj209 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj210 = obj250 ']' 2024-04-25T09:14:59.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj210 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj211 = obj250 ']' 2024-04-25T09:14:59.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj211 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj212 = obj250 ']' 2024-04-25T09:14:59.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj212 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj213 = obj250 ']' 2024-04-25T09:14:59.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj213 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:14:59.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:14:59.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj214 = obj250 ']' 2024-04-25T09:14:59.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj214 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj215 = obj250 ']' 2024-04-25T09:15:00.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj215 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj216 = obj250 ']' 2024-04-25T09:15:00.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj216 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj217 = obj250 ']' 2024-04-25T09:15:00.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj217 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj218 = obj250 ']' 2024-04-25T09:15:00.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj218 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj219 = obj250 ']' 2024-04-25T09:15:00.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj219 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj220 = obj250 ']' 2024-04-25T09:15:00.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj220 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj221 = obj250 ']' 2024-04-25T09:15:00.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj221 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj222 = obj250 ']' 2024-04-25T09:15:00.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj222 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj223 = obj250 ']' 2024-04-25T09:15:00.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj223 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj224 = obj250 ']' 2024-04-25T09:15:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj224 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj225 = obj250 ']' 2024-04-25T09:15:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj225 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj226 = obj250 ']' 2024-04-25T09:15:00.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj226 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj227 = obj250 ']' 2024-04-25T09:15:00.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj227 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj228 = obj250 ']' 2024-04-25T09:15:00.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj228 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj229 = obj250 ']' 2024-04-25T09:15:00.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj229 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj230 = obj250 ']' 2024-04-25T09:15:00.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj230 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.476 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj231 = obj250 ']' 2024-04-25T09:15:00.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj231 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj232 = obj250 ']' 2024-04-25T09:15:00.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj232 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj233 = obj250 ']' 2024-04-25T09:15:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj233 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj234 = obj250 ']' 2024-04-25T09:15:00.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj234 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj235 = obj250 ']' 2024-04-25T09:15:00.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj235 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj236 = obj250 ']' 2024-04-25T09:15:00.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj236 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj237 = obj250 ']' 2024-04-25T09:15:00.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj237 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.685 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj238 = obj250 ']' 2024-04-25T09:15:00.686 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj238 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj239 = obj250 ']' 2024-04-25T09:15:00.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj239 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj240 = obj250 ']' 2024-04-25T09:15:00.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj240 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.774 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj241 = obj250 ']' 2024-04-25T09:15:00.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj241 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj242 = obj250 ']' 2024-04-25T09:15:00.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj242 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj243 = obj250 ']' 2024-04-25T09:15:00.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj243 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj244 = obj250 ']' 2024-04-25T09:15:00.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj244 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj245 = obj250 ']' 2024-04-25T09:15:00.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj245 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj246 = obj250 ']' 2024-04-25T09:15:00.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj246 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj247 = obj250 ']' 2024-04-25T09:15:00.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj247 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:00.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:00.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj248 = obj250 ']' 2024-04-25T09:15:00.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj248 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj249 = obj250 ']' 2024-04-25T09:15:01.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj249 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj250 = obj250 ']' 2024-04-25T09:15:01.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:329: TEST_rep_backfill_unfound: rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.069 INFO:tasks.workunit.client.0.smithi102.stderr:error getting test-pool/obj250: (2) No such file or directory 2024-04-25T09:15:01.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj251 = obj250 ']' 2024-04-25T09:15:01.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj251 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj252 = obj250 ']' 2024-04-25T09:15:01.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj252 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj253 = obj250 ']' 2024-04-25T09:15:01.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj253 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj254 = obj250 ']' 2024-04-25T09:15:01.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj254 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj255 = obj250 ']' 2024-04-25T09:15:01.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj255 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj256 = obj250 ']' 2024-04-25T09:15:01.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj256 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj257 = obj250 ']' 2024-04-25T09:15:01.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj257 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj258 = obj250 ']' 2024-04-25T09:15:01.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj258 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj259 = obj250 ']' 2024-04-25T09:15:01.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj259 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj260 = obj250 ']' 2024-04-25T09:15:01.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj260 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj261 = obj250 ']' 2024-04-25T09:15:01.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj261 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj262 = obj250 ']' 2024-04-25T09:15:01.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj262 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj263 = obj250 ']' 2024-04-25T09:15:01.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj263 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj264 = obj250 ']' 2024-04-25T09:15:01.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj264 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj265 = obj250 ']' 2024-04-25T09:15:01.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj265 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj266 = obj250 ']' 2024-04-25T09:15:01.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj266 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj267 = obj250 ']' 2024-04-25T09:15:01.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj267 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj268 = obj250 ']' 2024-04-25T09:15:01.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj268 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj269 = obj250 ']' 2024-04-25T09:15:01.622 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj269 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj270 = obj250 ']' 2024-04-25T09:15:01.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj270 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj271 = obj250 ']' 2024-04-25T09:15:01.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj271 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj272 = obj250 ']' 2024-04-25T09:15:01.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj272 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj273 = obj250 ']' 2024-04-25T09:15:01.771 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj273 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj274 = obj250 ']' 2024-04-25T09:15:01.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj274 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj275 = obj250 ']' 2024-04-25T09:15:01.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj275 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj276 = obj250 ']' 2024-04-25T09:15:01.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj276 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj277 = obj250 ']' 2024-04-25T09:15:01.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj277 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj278 = obj250 ']' 2024-04-25T09:15:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj278 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj279 = obj250 ']' 2024-04-25T09:15:01.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj279 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj280 = obj250 ']' 2024-04-25T09:15:02.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj280 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj281 = obj250 ']' 2024-04-25T09:15:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj281 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj282 = obj250 ']' 2024-04-25T09:15:02.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj282 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj283 = obj250 ']' 2024-04-25T09:15:02.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj283 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj284 = obj250 ']' 2024-04-25T09:15:02.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj284 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj285 = obj250 ']' 2024-04-25T09:15:02.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj285 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj286 = obj250 ']' 2024-04-25T09:15:02.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj286 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj287 = obj250 ']' 2024-04-25T09:15:02.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj287 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj288 = obj250 ']' 2024-04-25T09:15:02.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj288 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj289 = obj250 ']' 2024-04-25T09:15:02.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj289 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.300 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj290 = obj250 ']' 2024-04-25T09:15:02.301 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj290 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj291 = obj250 ']' 2024-04-25T09:15:02.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj291 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj292 = obj250 ']' 2024-04-25T09:15:02.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj292 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj293 = obj250 ']' 2024-04-25T09:15:02.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj293 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj294 = obj250 ']' 2024-04-25T09:15:02.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj294 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj295 = obj250 ']' 2024-04-25T09:15:02.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj295 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj296 = obj250 ']' 2024-04-25T09:15:02.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj296 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj297 = obj250 ']' 2024-04-25T09:15:02.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj297 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj298 = obj250 ']' 2024-04-25T09:15:02.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj298 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj299 = obj250 ']' 2024-04-25T09:15:02.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj299 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:15:02.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj300 = obj250 ']' 2024-04-25T09:15:02.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj300 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:336: TEST_rep_backfill_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:15:02.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:338: TEST_rep_backfill_unfound: delete_pool test-pool 2024-04-25T09:15:02.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-04-25T09:15:02.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-04-25T09:15:03.109 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test-pool' does not exist 2024-04-25T09:15:03.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:15:03.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:15:03.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:15:03.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:15:03.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:15:03.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:15:03.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:15:03.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:15:03.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:15:03.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:15:03.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:15:03.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:15:03.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:15:03.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:15:03.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:15:03.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:15:03.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:15:03.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:15:03.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:15:03.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:15:03.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:15:03.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:15:03.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:15:03.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:15:03.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:03.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:15:03.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.388535 2024-04-25T09:15:03.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:15:03.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:15:03.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T09:15:03.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T09:15:03.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:15:03.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:15:03.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:15:03.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:15:03.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:15:03.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:15:03.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:15:03.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:15:03.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:15:03.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:15:03.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:15:03.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:15:03.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:15:03.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:15:03.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:15:03.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:15:03.497 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:15:03.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:15:03.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:15:03.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:15:03.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:15:03.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:15:03.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:15:03.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:15:03.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:15:03.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:03.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:15:03.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.388535 2024-04-25T09:15:03.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:15:03.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:15:03.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T09:15:03.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:15:03.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:03.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:15:03.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.388535 2024-04-25T09:15:03.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:15:03.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:15:03.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:15:03.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T09:15:03.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T09:15:03.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:15:03.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:15:03.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:15:03.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:15:03.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T09:15:03.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T09:15:04.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:15:04.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:15:04.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:15:04.013 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:15:04.013 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:04.013 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:15:04.013 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:15:04.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T09:15:04.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:15:04.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:15:04.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:15:04.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:15:04.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:15:04.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:15:04.248 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:15:04.248 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:15:04.249 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:15:04.249 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:15:04.250 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:04.250 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:15:04.250 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-mon.a.asok 2024-04-25T09:15:04.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:15:04.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.388535/ceph-mon.a.asok config get fsid 2024-04-25T09:15:04.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:15:04.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:15:04.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:15:04.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:15:04.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:15:04.364 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:15:04.364 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:15:04.365 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:15:04.365 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:15:04.366 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:04.366 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:15:04.366 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-mon.a.asok 2024-04-25T09:15:04.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:15:04.366 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.388535/ceph-mon.a.asok config get mon_host 2024-04-25T09:15:04.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T09:15:04.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:15:04.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:15:04.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:15:04.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:15:04.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T09:15:04.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:15:05.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:15:05.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:15:05.394 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:15:05.395 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:15:05.395 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:05.395 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:15:05.395 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:15:05.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:15:05.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:15:05.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T09:15:05.743 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T09:15:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_recovery_unfound td/osd-rep-recov-eio.sh 2024-04-25T09:15:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:343: TEST_rep_recovery_unfound: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:15:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:344: TEST_rep_recovery_unfound: local objname=myobject 2024-04-25T09:15:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:345: TEST_rep_recovery_unfound: local lastobj=100 2024-04-25T09:15:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:347: TEST_rep_recovery_unfound: local testobj=obj75 2024-04-25T09:15:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:349: TEST_rep_recovery_unfound: setup_osds 3 2024-04-25T09:15:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-04-25T09:15:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T09:15:05.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T09:15:05.761 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-04-25T09:15:05.762 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-04-25T09:15:05.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:15:05.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:15:05.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:15:05.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:15:05.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:15:05.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:15:05.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:15:05.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:15:05.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:15:05.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:15:05.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:15:05.769 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:15:05.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5a480d30-6510-4430-a563-20017f392e4f 2024-04-25T09:15:05.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5a480d30-6510-4430-a563-20017f392e4f' 2024-04-25T09:15:05.770 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 5a480d30-6510-4430-a563-20017f392e4f 2024-04-25T09:15:05.770 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:15:05.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZHypm+O7wLxAAtDegrhTcHDYc78gCqgRZVw== 2024-04-25T09:15:05.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZHypm+O7wLxAAtDegrhTcHDYc78gCqgRZVw=="}' 2024-04-25T09:15:05.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5a480d30-6510-4430-a563-20017f392e4f -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:15:06.120 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:15:06.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:15:06.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZHypm+O7wLxAAtDegrhTcHDYc78gCqgRZVw== --osd-uuid 5a480d30-6510-4430-a563-20017f392e4f 2024-04-25T09:15:06.166 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:06.178+0000 7fa0bdd2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:06.197 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:06.210+0000 7fa0bdd2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:06.198 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:06.210+0000 7fa0bdd2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:06.199 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:06.210+0000 7fa0bdd2f7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:15:06.199 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:06.210+0000 7fa0bdd2f7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:15:06.199 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:06.210+0000 7fa0bdd2f7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:15:06.236 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:06.250+0000 7fa0bdd2f7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T09:15:12.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T09:15:12.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:15:12.844 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:15:12.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:15:12.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:15:13.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:15:13.406 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:15:13.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:15:13.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:15:13.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:15:13.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:15:13.440 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:13.454+0000 7f994d1607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:13.442 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:13.454+0000 7f994d1607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:13.444 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:13.454+0000 7f994d1607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:15:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:15:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:15:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:15:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:15:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:13.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:15:13.817 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:15:13.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:13.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:14.018 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:14.030+0000 7f994d1607c0 -1 Falling back to public interface 2024-04-25T09:15:14.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:15.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:15.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:15.267 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:15:15.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:15:15.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:15.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:15.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:16.670 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:16.682+0000 7f994d1607c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:15:16.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:16.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:16.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:15:16.693 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:15:16.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:16.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:17.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:18.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:18.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:18.248 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:15:18.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:15:18.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:18.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:18.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:19.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:19.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:19.709 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:15:19.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:15:19.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:19.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:20.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:21.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:21.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:21.197 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:15:21.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:15:21.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:21.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:21.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:22.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:22.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:22.707 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:15:22.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:15:22.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:22.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:23.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:24.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:24.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:24.179 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:15:24.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:15:24.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:24.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:24.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:25.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:25.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:25.655 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:15:25.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:15:25.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:25.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:26.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:27.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:27.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:27.087 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:15:27.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:15:27.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:27.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:27.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:28.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:28.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:28.613 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:15:28.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:15:28.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:28.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:29.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:30.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:30.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:30.030 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:15:30.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:15:30.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:30.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:30.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:31.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:31.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:31.519 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:15:31.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:15:31.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:31.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:32.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:33.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:33.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:33.434 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:15:33.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:15:33.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:33.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:34.368 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:35.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:35.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:35.371 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:15:35.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:15:35.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:35.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:36.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:37.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:37.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:37.257 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:15:37.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:15:37.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:37.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:38.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:39.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:39.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:39.046 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:15:39.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:15:39.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:39.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:40.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:41.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:41.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:41.020 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:15:41.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:15:41.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:41.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:41.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:42.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:42.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:42.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:15:42.815 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:15:42.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:42.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:43.409 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/849178726,v1:127.0.0.1:6803/849178726] [v2:127.0.0.1:6804/849178726,v1:127.0.0.1:6805/849178726] exists,up 5a480d30-6510-4430-a563-20017f392e4f 2024-04-25T09:15:43.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:15:43.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:15:43.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:15:43.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:15:43.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:15:43.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:15:43.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:15:43.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:15:43.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:15:43.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:15:43.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:15:43.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:15:43.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:15:43.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T09:15:43.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:15:43.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=94208511-4eb6-4b42-a176-a941da7fceae 2024-04-25T09:15:43.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 94208511-4eb6-4b42-a176-a941da7fceae' 2024-04-25T09:15:43.417 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 94208511-4eb6-4b42-a176-a941da7fceae 2024-04-25T09:15:43.417 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:15:43.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/HypmGdsUGxAA3iv+XqptgXgzKsieY2nOaA== 2024-04-25T09:15:43.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/HypmGdsUGxAA3iv+XqptgXgzKsieY2nOaA=="}' 2024-04-25T09:15:43.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 94208511-4eb6-4b42-a176-a941da7fceae -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:15:44.146 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:15:44.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:15:44.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/HypmGdsUGxAA3iv+XqptgXgzKsieY2nOaA== --osd-uuid 94208511-4eb6-4b42-a176-a941da7fceae 2024-04-25T09:15:44.199 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:44.210+0000 7f7e295e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:44.212 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:44.222+0000 7f7e295e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:44.213 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:44.226+0000 7f7e295e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:44.213 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:44.226+0000 7f7e295e27c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:15:44.213 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:44.226+0000 7f7e295e27c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:15:44.214 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:44.226+0000 7f7e295e27c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:15:44.346 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:44.358+0000 7f7e295e27c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T09:15:51.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T09:15:51.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:15:51.009 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:15:51.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:15:51.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:15:51.572 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:15:51.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:15:51.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:15:51.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:15:51.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:15:51.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:15:51.605 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:51.617+0000 7f5c3eaef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:51.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:51.617+0000 7f5c3eaef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:51.608 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:51.621+0000 7f5c3eaef7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:51.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:15:51.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:15:51.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:15:51.981 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:15:51.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:15:51.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:15:51.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:51.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:15:51.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:51.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:52.153 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:52.165+0000 7f5c3eaef7c0 -1 Falling back to public interface 2024-04-25T09:15:52.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:53.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:53.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:53.411 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:15:53.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:15:53.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:53.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:53.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:54.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:54.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:54.836 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:15:54.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:15:54.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:54.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:54.911 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:15:54.921+0000 7f5c3eaef7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:15:55.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:56.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:56.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:56.290 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:15:56.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:15:56.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:56.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:56.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:57.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:57.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:57.822 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:15:57.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:15:57.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:57.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:58.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:59.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:59.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:59.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:15:59.277 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:15:59.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:59.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:59.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:00.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:00.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:00.829 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:16:00.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:16:00.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:00.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:01.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:02.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:02.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:02.260 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:16:02.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:16:02.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:02.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:02.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:03.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:03.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:03.780 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:16:03.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:16:03.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:03.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:04.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:05.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:05.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:05.204 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:16:05.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:16:05.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:05.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:05.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:06.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:06.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:06.631 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:16:06.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:16:06.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:06.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:07.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:08.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:08.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:08.216 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:16:08.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:16:08.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:08.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:08.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:09.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:09.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:09.801 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:16:09.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:16:09.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:09.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:10.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:11.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:11.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:11.524 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:16:11.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:16:11.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:11.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:12.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:13.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:13.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:13.354 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:16:13.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:16:13.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:13.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:14.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:15.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:15.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:15.228 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:16:15.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:16:15.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:15.229 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:16.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:17.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:17.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:17.428 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:16:17.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:16:17.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:17.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:18.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:19.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:19.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:19.295 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:16:19.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:16:19.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:19.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:20.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:21.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:21.216 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:21.216 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:16:21.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:16:21.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:21.217 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:21.897 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3742820052,v1:127.0.0.1:6811/3742820052] [v2:127.0.0.1:6812/3742820052,v1:127.0.0.1:6813/3742820052] exists,up 94208511-4eb6-4b42-a176-a941da7fceae 2024-04-25T09:16:21.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:16:21.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:16:21.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:16:21.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:16:21.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:16:21.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:16:21.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:16:21.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:16:21.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:16:21.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:16:21.901 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:16:21.901 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:21.901 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:16:21.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:16:21.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:16:21.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:16:21.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c111db78-7935-4b6a-b125-67d13dfeaf87 2024-04-25T09:16:21.904 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 c111db78-7935-4b6a-b125-67d13dfeaf87 2024-04-25T09:16:21.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c111db78-7935-4b6a-b125-67d13dfeaf87' 2024-04-25T09:16:21.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:16:21.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlHypmVuE0OBAARBy2XQ7Nr9u4nwKFMVvcKQ== 2024-04-25T09:16:21.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlHypmVuE0OBAARBy2XQ7Nr9u4nwKFMVvcKQ=="}' 2024-04-25T09:16:21.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c111db78-7935-4b6a-b125-67d13dfeaf87 -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:16:22.707 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:16:22.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:16:22.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlHypmVuE0OBAARBy2XQ7Nr9u4nwKFMVvcKQ== --osd-uuid c111db78-7935-4b6a-b125-67d13dfeaf87 2024-04-25T09:16:22.754 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:22.765+0000 7fb2e896b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:22.787 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:22.797+0000 7fb2e896b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:22.789 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:22.801+0000 7fb2e896b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:22.789 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:22.801+0000 7fb2e896b7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:16:22.789 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:22.801+0000 7fb2e896b7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:16:22.790 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:22.801+0000 7fb2e896b7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:16:22.921 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:22.933+0000 7fb2e896b7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:16:29.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:16:29.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:16:29.658 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T09:16:29.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:16:29.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:16:30.202 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:16:30.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:16:30.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:16:30.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:16:30.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:16:30.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:16:30.237 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:30.249+0000 7f3eea33e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:30.239 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:30.249+0000 7f3eea33e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:30.240 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:30.253+0000 7f3eea33e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:30.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:16:30.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:16:30.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:16:30.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:16:30.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:16:30.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:30.636 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:16:30.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:16:30.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:30.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:30.797 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:30.809+0000 7f3eea33e7c0 -1 Falling back to public interface 2024-04-25T09:16:31.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:32.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:32.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:32.064 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:16:32.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:16:32.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:32.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:32.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:33.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:33.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:33.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:16:33.471 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:16:33.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:33.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:33.474 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:16:33.485+0000 7f3eea33e7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:16:33.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:34.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:34.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:34.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:16:34.882 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:16:34.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:34.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:35.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:36.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:36.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:36.471 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:16:36.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:16:36.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:36.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:37.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:38.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:38.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:38.009 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:16:38.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:16:38.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:38.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:38.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:39.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:39.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:39.556 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:16:39.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:16:39.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:39.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:39.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:40.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:40.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:40.992 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:16:40.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:16:40.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:40.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:41.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:42.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:42.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:42.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:16:42.568 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:16:42.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:42.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:43.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:44.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:44.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:44.100 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:16:44.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:16:44.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:44.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:44.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:45.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:45.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:45.640 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:16:45.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:16:45.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:45.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:46.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:47.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:47.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:47.103 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:16:47.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:16:47.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:47.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:47.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:48.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:48.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:48.581 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:16:48.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:16:48.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:48.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:49.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:50.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:50.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:50.296 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:16:50.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:16:50.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:50.296 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:50.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:51.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:51.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:51.775 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:16:51.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:16:51.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:51.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:52.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:53.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:53.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:53.361 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:16:53.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:16:53.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:53.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:54.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:55.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:55.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:55.263 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:16:55.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:16:55.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:55.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:56.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:57.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:57.075 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:16:57.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:57.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:16:57.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:57.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:57.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:58.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:58.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:58.743 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:16:58.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:16:58.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:58.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:59.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:00.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:00.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:00.940 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T09:17:00.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:17:00.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:00.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:17:01.575 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3354025060,v1:127.0.0.1:6819/3354025060] [v2:127.0.0.1:6820/3354025060,v1:127.0.0.1:6821/3354025060] exists,up c111db78-7935-4b6a-b125-67d13dfeaf87 2024-04-25T09:17:01.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:17:01.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:17:01.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:17:01.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:17:01.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:17:01.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:17:01.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:17:01.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:17:01.577 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:17:01.578 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:17:01.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:17:01.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:17:01.578 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:17:01.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:17:01.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:17:01.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:17:01.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:17:01.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:17:01.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:17:02.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:17:02.303 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:17:02.304 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:17:02.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:17:02.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:02.304 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:17:02.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803798 2024-04-25T09:17:02.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803798 2024-04-25T09:17:02.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803798' 2024-04-25T09:17:02.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:02.490 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:17:02.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574860 2024-04-25T09:17:02.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574860 2024-04-25T09:17:02.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803798 1-55834574860' 2024-04-25T09:17:02.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:02.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:17:02.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345931 2024-04-25T09:17:02.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345931 2024-04-25T09:17:02.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803798 1-55834574860 2-85899345931' 2024-04-25T09:17:02.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:02.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803798 2024-04-25T09:17:02.808 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:02.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:17:02.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803798 2024-04-25T09:17:02.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:02.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803798 2024-04-25T09:17:02.811 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803798 2024-04-25T09:17:02.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803798' 2024-04-25T09:17:02.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:17:03.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803796 -lt 25769803798 2024-04-25T09:17:03.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:17:04.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:17:04.227 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:17:04.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803799 -lt 25769803798 2024-04-25T09:17:04.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:04.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574860 2024-04-25T09:17:04.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:04.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:17:04.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574860 2024-04-25T09:17:04.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:04.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574860 2024-04-25T09:17:04.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574860' 2024-04-25T09:17:04.671 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574860 2024-04-25T09:17:04.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:17:05.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574860 -lt 55834574860 2024-04-25T09:17:05.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:05.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345931 2024-04-25T09:17:05.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:05.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:17:05.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345931 2024-04-25T09:17:05.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:05.109 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345931 2024-04-25T09:17:05.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345931 2024-04-25T09:17:05.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345931' 2024-04-25T09:17:05.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:17:05.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345931 -lt 85899345931 2024-04-25T09:17:05.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:17:05.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:05.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:06.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:17:06.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:17:06.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:17:06.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:17:06.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:17:06.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:17:06.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:17:06.107 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:17:06.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:17:06.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:17:06.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:06.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:07.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:17:07.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:17:07.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:17:07.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:351: TEST_rep_recovery_unfound: local poolname=test-pool 2024-04-25T09:17:07.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:352: TEST_rep_recovery_unfound: create_pool test-pool 1 1 2024-04-25T09:17:07.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-04-25T09:17:07.618 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test-pool' already exists 2024-04-25T09:17:07.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:17:08.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:353: TEST_rep_recovery_unfound: wait_for_clean 2024-04-25T09:17:08.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:17:08.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:17:08.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:17:08.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:17:08.639 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:17:08.639 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:17:08.640 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:17:08.640 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:17:08.640 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:17:08.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:17:08.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:17:08.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:17:08.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:17:08.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:17:08.766 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:17:09.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:17:09.199 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:17:09.199 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:17:09.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:17:09.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:09.200 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:17:09.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803802 2024-04-25T09:17:09.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803802 2024-04-25T09:17:09.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803802' 2024-04-25T09:17:09.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:09.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:17:09.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574863 2024-04-25T09:17:09.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574863 2024-04-25T09:17:09.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803802 1-55834574863' 2024-04-25T09:17:09.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:09.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:17:09.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345934 2024-04-25T09:17:09.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345934 2024-04-25T09:17:09.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803802 1-55834574863 2-85899345934' 2024-04-25T09:17:09.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:09.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803802 2024-04-25T09:17:09.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:09.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:17:09.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803802 2024-04-25T09:17:09.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:09.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803802 2024-04-25T09:17:09.761 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 25769803802 2024-04-25T09:17:09.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803802' 2024-04-25T09:17:09.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:17:10.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803802 -lt 25769803802 2024-04-25T09:17:10.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:10.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574863 2024-04-25T09:17:10.188 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:10.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:17:10.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574863 2024-04-25T09:17:10.192 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:10.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574863 2024-04-25T09:17:10.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574863' 2024-04-25T09:17:10.193 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574863 2024-04-25T09:17:10.193 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:17:10.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574863 -lt 55834574863 2024-04-25T09:17:10.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:10.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:10.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345934 2024-04-25T09:17:10.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:17:10.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345934 2024-04-25T09:17:10.696 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:10.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345934 2024-04-25T09:17:10.698 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345934 2024-04-25T09:17:10.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345934' 2024-04-25T09:17:10.698 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:17:11.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345934 -lt 85899345934 2024-04-25T09:17:11.134 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:17:11.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:11.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:11.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:17:11.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:17:11.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:17:11.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:17:11.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:17:11.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:17:11.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:17:11.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:17:12.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:17:12.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:17:12.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:12.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:12.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:17:12.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:17:12.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:17:12.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:355: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:17:13.038 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:17:13.038 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.173403+0000 0'0 25:34 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:35:37.681354+0000 0 0 2024-04-25T09:17:13.038 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.174916+0000 0'0 25:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:48:22.123869+0000 0 0 2024-04-25T09:17:13.038 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.174898+0000 0'0 26:20 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:06:11.705081+0000 0 0 2024-04-25T09:17:13.038 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.173166+0000 0'0 25:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:46:09.069947+0000 0 0 2024-04-25T09:17:13.038 INFO:tasks.workunit.client.0.smithi102.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:09.399426+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:17:07.433555+0000 0'0 2024-04-25T09:17:07.433555+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:34:34.158139+0000 0 0 2024-04-25T09:17:13.039 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.172758+0000 0'0 25:34 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:37:23.419609+0000 0 0 2024-04-25T09:17:13.039 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:01.789756+0000 0'0 25:50 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:11:49.961249+0000 0 0 2024-04-25T09:17:13.039 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.172914+0000 0'0 26:20 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:10:26.104977+0000 0 0 2024-04-25T09:17:13.039 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.174943+0000 0'0 25:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:47:44.556345+0000 0 0 2024-04-25T09:17:13.039 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T09:17:13.039 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:17:13.039 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T09:17:13.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:357: TEST_rep_recovery_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-04-25T09:17:13.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:13.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-04-25T09:17:13.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-04-25T09:17:13.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:13.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:17:13.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:13.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:17:13.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:13.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:17:13.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:13.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:17:13.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:13.222 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: get_osds test-pool myobject 2024-04-25T09:17:13.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:17:13.223 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-25T09:17:13.223 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-04-25T09:17:13.223 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:17:13.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:17:13.652 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:17:13.652 INFO:tasks.workunit.client.0.smithi102.stderr:0' 2024-04-25T09:17:13.652 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-25T09:17:13.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: initial_osds=('2' '1' '0') 2024-04-25T09:17:13.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: local -a initial_osds 2024-04-25T09:17:13.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:360: TEST_rep_recovery_unfound: local last_osd=0 2024-04-25T09:17:13.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:361: TEST_rep_recovery_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-04-25T09:17:13.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:17:13.653 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:17:13.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:17:13.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:17:13.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:17:13.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:17:13.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:362: TEST_rep_recovery_unfound: ceph osd down 0 2024-04-25T09:17:14.761 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already down. 2024-04-25T09:17:14.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:363: TEST_rep_recovery_unfound: ceph osd out 0 2024-04-25T09:17:15.377 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. 2024-04-25T09:17:15.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:365: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:17:15.819 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:17:15.819 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.173403+0000 0'0 26:36 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:35:37.681354+0000 0 0 2024-04-25T09:17:15.820 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.174916+0000 0'0 26:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:48:22.123869+0000 0 0 2024-04-25T09:17:15.820 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.174898+0000 0'0 26:20 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:06:11.705081+0000 0 0 2024-04-25T09:17:15.820 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.173166+0000 0'0 26:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:46:09.069947+0000 0 0 2024-04-25T09:17:15.820 INFO:tasks.workunit.client.0.smithi102.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:09.399426+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:17:07.433555+0000 0'0 2024-04-25T09:17:07.433555+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:34:34.158139+0000 0 0 2024-04-25T09:17:15.820 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.172758+0000 0'0 26:36 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:37:23.419609+0000 0 0 2024-04-25T09:17:15.820 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2024-04-25T09:17:01.789756+0000 0'0 26:52 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:11:49.961249+0000 0 0 2024-04-25T09:17:15.820 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.172914+0000 0'0 26:20 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:10:26.104977+0000 0 0 2024-04-25T09:17:15.820 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:02.174943+0000 0'0 26:36 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:47:44.556345+0000 0 0 2024-04-25T09:17:15.820 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T09:17:15.820 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:17:15.820 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T09:17:15.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:367: TEST_rep_recovery_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-04-25T09:17:15.836 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records in 2024-04-25T09:17:15.836 INFO:tasks.workunit.client.0.smithi102.stderr:4+0 records out 2024-04-25T09:17:15.836 INFO:tasks.workunit.client.0.smithi102.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000178148 s, 23.0 MB/s 2024-04-25T09:17:15.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:17:15.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:15.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:15.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:15.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:16.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:16.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:16.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:16.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:16.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:16.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:16.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:16.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:16.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:16.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:17.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:17.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:17.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:17.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:17.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:17.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:17.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:17.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:17.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:17.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:17.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:17.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:18.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:18.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:18.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:18.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:18.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:18.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:18.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:18.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:18.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:18.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:18.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:19.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:19.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:19.231 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:19.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:19.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:19.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:19.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:19.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:19.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:19.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:19.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:19.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:20.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:20.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:20.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:20.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:20.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:20.437 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:20.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:20.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:20.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:20.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:20.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:20.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:21.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:21.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:21.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:21.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:21.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:21.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:21.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:21.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:21.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:21.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:21.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:21.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:22.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:22.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:22.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:22.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:22.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:22.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:22.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:22.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:22.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:22.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:22.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:22.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:23.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:23.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:23.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:23.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:23.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:23.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:23.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:23.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:23.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:23.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:23.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:24.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:24.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:24.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:24.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:24.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:24.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:24.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:24.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:24.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:24.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:24.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:24.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:25.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:25.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:25.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:25.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:25.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:25.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:25.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:25.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:25.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:25.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:25.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:25.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:26.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:26.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:26.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:26.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:26.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:26.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:26.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:26.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:26.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:26.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:26.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:26.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:27.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:27.148 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:27.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:27.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:27.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:27.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:27.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:27.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:27.849 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:28.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:28.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:28.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:28.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:28.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:28.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:28.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:28.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:28.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:28.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:28.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:28.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:29.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:29.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:29.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:29.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:29.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:29.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:29.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:29.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:29.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:29.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:29.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:29.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:29.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:30.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:30.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:30.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:30.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:30.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:30.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:30.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:30.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:30.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:30.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:30.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:31.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:31.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:31.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:31.230 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:31.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:31.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:31.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:31.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:31.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:31.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:31.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:31.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:32.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:32.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:32.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:32.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:32.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:32.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:32.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:17:32.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:32.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:373: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 0 2024-04-25T09:17:32.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:17:32.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:17:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:17:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:17:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:17:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:17:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj75 2024-04-25T09:17:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:17:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:17:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:17:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:17:32.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj75 2024-04-25T09:17:32.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:17:32.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-04-25T09:17:32.813 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:17:32.813 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-04-25T09:17:33.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:17:33.247 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-04-25T09:17:33.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:17:33.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:17:33.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:17:33.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=2 2024-04-25T09:17:33.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:17:33.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:17:33.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-04-25T09:17:33.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:17:33.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-04-25T09:17:33.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:17:33.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-04-25T09:17:33.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:17:33.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:17:33.249 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:17:33.249 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-04-25T09:17:33.249 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:17:33.249 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:17:33.250 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:33.250 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:33.251 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:17:33.251 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.2.asok 2024-04-25T09:17:33.251 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:17:33.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:17:33.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:17:33.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:17:33.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.2 2024-04-25T09:17:33.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:17:33.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:17:33.373 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:33.373 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:33.373 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:17:33.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.2.asok 2024-04-25T09:17:33.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:17:33.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.2.asok injectdataerr test-pool obj75 2024-04-25T09:17:33.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:374: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 1 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj75 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj75 2024-04-25T09:17:33.494 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:17:33.495 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-04-25T09:17:33.495 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-04-25T09:17:33.495 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:17:33.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:17:33.925 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-04-25T09:17:33.925 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:17:33.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:17:33.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:17:33.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:17:33.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:17:33.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:17:33.926 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:17:33.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:17:33.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:17:33.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:17:33.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:17:33.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:17:33.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:17:33.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:17:33.928 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:17:33.928 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:17:33.928 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:17:33.929 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:33.929 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:33.929 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:17:33.929 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:17:33.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.388535/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:17:34.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:17:34.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:17:34.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:17:34.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:17:34.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:17:34.043 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:17:34.043 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:34.044 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:34.044 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:17:34.044 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.388535/ceph-osd.1.asok 2024-04-25T09:17:34.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:17:34.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.388535/ceph-osd.1.asok injectdataerr test-pool obj75 2024-04-25T09:17:34.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:376: TEST_rep_recovery_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:17:34.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:34.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:17:34.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:17:34.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:17:34.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:17:34.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:17:34.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:17:34.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:17:34.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' 2024-04-25T09:17:34.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:17:34.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:17:34.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:17:34.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:17:34.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:17:34.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:17:34.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:17:34.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:17:34.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:17:34.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:17:34.157 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:17:34.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:17:34.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=3ad78387-2d96-4ac5-b314-cb95cfa29968 --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.388535/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:17:34.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-04-25T09:17:34.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:17:34.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:17:34.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:17:34.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:17:34.195 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:17:34.207+0000 7fdd7374f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:34.203 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:17:34.215+0000 7fdd7374f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:34.204 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:17:34.219+0000 7fdd7374f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:34.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:17:34.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:17:34.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:17:34.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:17:34.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:17:34.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:34.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:17:34.572 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:17:34.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:34.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:34.749 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:17:34.763+0000 7fdd7374f7c0 -1 Falling back to public interface 2024-04-25T09:17:35.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:36.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:36.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:36.006 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:17:36.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:17:36.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:36.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:36.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:37.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:37.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:37.426 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:17:37.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:17:37.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:37.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:37.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:38.231 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:17:38.243+0000 7fdd7374f7c0 -1 osd.0 26 log_to_monitors true 2024-04-25T09:17:38.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:38.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:38.850 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:17:38.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:17:38.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:38.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:17:39.495 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up out weight 0 up_from 32 up_thru 20 down_at 27 last_clean_interval [6,26) [v2:127.0.0.1:6802/2037682632,v1:127.0.0.1:6803/2037682632] [v2:127.0.0.1:6804/2037682632,v1:127.0.0.1:6805/2037682632] exists,up 5a480d30-6510-4430-a563-20017f392e4f 2024-04-25T09:17:39.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:17:39.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:17:39.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:17:39.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:377: TEST_rep_recovery_unfound: ceph osd in 0 2024-04-25T09:17:41.133 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already in. 2024-04-25T09:17:41.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:379: TEST_rep_recovery_unfound: sleep 15 2024-04-25T09:17:56.156 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:17:56.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2024-04-25T09:17:56.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2024-04-25T09:17:56.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:17:56.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:17:56.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:17:56.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:17:56.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovery_unfound+degraded 2024-04-25T09:17:56.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovery_unfound+degraded 2024-04-25T09:17:56.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2024-04-25T09:17:56.571 INFO:tasks.workunit.client.0.smithi102.stdout:active+recovery_unfound+degraded 2024-04-25T09:17:56.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 0 = 0 ']' 2024-04-25T09:17:56.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:385: TEST_rep_recovery_unfound: break 2024-04-25T09:17:56.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:391: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:17:56.949 INFO:tasks.workunit.client.0.smithi102.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:17:56.949 INFO:tasks.workunit.client.0.smithi102.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:42.013340+0000 0'0 40:69 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:20:03.738287+0000 0 0 2024-04-25T09:17:56.949 INFO:tasks.workunit.client.0.smithi102.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:42.873022+0000 0'0 40:68 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:57:39.646063+0000 0 0 2024-04-25T09:17:56.949 INFO:tasks.workunit.client.0.smithi102.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:43.417461+0000 0'0 40:51 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:07:25.114671+0000 0 0 2024-04-25T09:17:56.949 INFO:tasks.workunit.client.0.smithi102.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:42.687174+0000 0'0 40:68 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:41:34.093955+0000 0 0 2024-04-25T09:17:56.949 INFO:tasks.workunit.client.0.smithi102.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-04-25T09:17:47.587069+0000 30'101 40:347 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:17:07.433555+0000 0'0 2024-04-25T09:17:07.433555+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:17:56.949 INFO:tasks.workunit.client.0.smithi102.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:42.012461+0000 0'0 40:69 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:01:44.488516+0000 0 0 2024-04-25T09:17:56.949 INFO:tasks.workunit.client.0.smithi102.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:42.687208+0000 0'0 40:56 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:17:32.711489+0000 0 0 2024-04-25T09:17:56.949 INFO:tasks.workunit.client.0.smithi102.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:43.417474+0000 0'0 40:58 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:31:22.898811+0000 0 0 2024-04-25T09:17:56.949 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:17:42.873375+0000 0'0 40:68 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:15:05.626629+0000 0'0 2024-04-25T09:15:05.626629+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:30:51.611492+0000 0 0 2024-04-25T09:17:56.949 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T09:17:56.950 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:17:56.950 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pgs 2024-04-25T09:17:56.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: grep -q obj75 2024-04-25T09:17:56.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: ceph pg 2.0 list_unfound 2024-04-25T09:17:57.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:395: TEST_rep_recovery_unfound: timeout 5 rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:02.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:396: TEST_rep_recovery_unfound: test 124 = 124 2024-04-25T09:18:02.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:398: TEST_rep_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-04-25T09:18:02.367 INFO:tasks.workunit.client.0.smithi102.stderr:pg has no unfound objects 2024-04-25T09:18:02.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:400: TEST_rep_recovery_unfound: wait_for_clean 2024-04-25T09:18:02.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:18:02.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:18:02.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:18:02.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:18:02.386 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:18:02.386 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:18:02.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:18:02.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:18:02.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:18:02.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:18:02.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:18:02.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:18:02.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:18:02.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:18:02.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:18:02.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:18:02.929 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:18:02.929 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:18:02.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:18:02.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:02.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:18:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953479 2024-04-25T09:18:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953479 2024-04-25T09:18:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953479' 2024-04-25T09:18:03.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:03.091 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:18:03.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574876 2024-04-25T09:18:03.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574876 2024-04-25T09:18:03.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953479 1-55834574876' 2024-04-25T09:18:03.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:03.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:18:03.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345947 2024-04-25T09:18:03.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345947 2024-04-25T09:18:03.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953479 1-55834574876 2-85899345947' 2024-04-25T09:18:03.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:03.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-137438953479 2024-04-25T09:18:03.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:03.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:18:03.409 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-137438953479 2024-04-25T09:18:03.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:03.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953479 2024-04-25T09:18:03.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 137438953479' 2024-04-25T09:18:03.411 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 137438953479 2024-04-25T09:18:03.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:03.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953477 -lt 137438953479 2024-04-25T09:18:03.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:18:04.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:18:04.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:05.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953479 -lt 137438953479 2024-04-25T09:18:05.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:05.253 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574876 2024-04-25T09:18:05.254 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:05.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:18:05.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574876 2024-04-25T09:18:05.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:05.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574876 2024-04-25T09:18:05.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574876' 2024-04-25T09:18:05.258 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 55834574876 2024-04-25T09:18:05.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:18:05.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574876 -lt 55834574876 2024-04-25T09:18:05.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:05.687 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345947 2024-04-25T09:18:05.688 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:05.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:18:05.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345947 2024-04-25T09:18:05.690 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:05.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345947 2024-04-25T09:18:05.691 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 85899345947 2024-04-25T09:18:05.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345947' 2024-04-25T09:18:05.691 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:18:06.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345947 -lt 85899345947 2024-04-25T09:18:06.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:18:06.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:18:06.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:18:06.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:18:06.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:18:06.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:18:06.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:18:06.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:18:06.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:18:06.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:18:06.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:18:07.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:18:07.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:18:07.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:18:07.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:18:07.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:18:07.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:18:07.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:18:07.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:18:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj1 = obj75 ']' 2024-04-25T09:18:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:07.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj2 = obj75 ']' 2024-04-25T09:18:07.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:07.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj3 = obj75 ']' 2024-04-25T09:18:07.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:07.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj4 = obj75 ']' 2024-04-25T09:18:07.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:07.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj5 = obj75 ']' 2024-04-25T09:18:07.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj6 = obj75 ']' 2024-04-25T09:18:07.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:07.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj7 = obj75 ']' 2024-04-25T09:18:07.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:07.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj8 = obj75 ']' 2024-04-25T09:18:07.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:07.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:07.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj9 = obj75 ']' 2024-04-25T09:18:07.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj10 = obj75 ']' 2024-04-25T09:18:08.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj11 = obj75 ']' 2024-04-25T09:18:08.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj12 = obj75 ']' 2024-04-25T09:18:08.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj13 = obj75 ']' 2024-04-25T09:18:08.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj14 = obj75 ']' 2024-04-25T09:18:08.137 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj15 = obj75 ']' 2024-04-25T09:18:08.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj16 = obj75 ']' 2024-04-25T09:18:08.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj17 = obj75 ']' 2024-04-25T09:18:08.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj18 = obj75 ']' 2024-04-25T09:18:08.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj19 = obj75 ']' 2024-04-25T09:18:08.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj20 = obj75 ']' 2024-04-25T09:18:08.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj21 = obj75 ']' 2024-04-25T09:18:08.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj22 = obj75 ']' 2024-04-25T09:18:08.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj23 = obj75 ']' 2024-04-25T09:18:08.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj24 = obj75 ']' 2024-04-25T09:18:08.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj25 = obj75 ']' 2024-04-25T09:18:08.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj26 = obj75 ']' 2024-04-25T09:18:08.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj27 = obj75 ']' 2024-04-25T09:18:08.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj28 = obj75 ']' 2024-04-25T09:18:08.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj29 = obj75 ']' 2024-04-25T09:18:08.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj30 = obj75 ']' 2024-04-25T09:18:08.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj31 = obj75 ']' 2024-04-25T09:18:08.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj32 = obj75 ']' 2024-04-25T09:18:08.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj33 = obj75 ']' 2024-04-25T09:18:08.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj34 = obj75 ']' 2024-04-25T09:18:08.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj35 = obj75 ']' 2024-04-25T09:18:08.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj36 = obj75 ']' 2024-04-25T09:18:08.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj37 = obj75 ']' 2024-04-25T09:18:08.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj38 = obj75 ']' 2024-04-25T09:18:08.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj39 = obj75 ']' 2024-04-25T09:18:08.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj40 = obj75 ']' 2024-04-25T09:18:08.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:08.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:08.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj41 = obj75 ']' 2024-04-25T09:18:08.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj42 = obj75 ']' 2024-04-25T09:18:09.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj43 = obj75 ']' 2024-04-25T09:18:09.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj44 = obj75 ']' 2024-04-25T09:18:09.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj45 = obj75 ']' 2024-04-25T09:18:09.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj46 = obj75 ']' 2024-04-25T09:18:09.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj47 = obj75 ']' 2024-04-25T09:18:09.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj48 = obj75 ']' 2024-04-25T09:18:09.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj49 = obj75 ']' 2024-04-25T09:18:09.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj50 = obj75 ']' 2024-04-25T09:18:09.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj51 = obj75 ']' 2024-04-25T09:18:09.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj52 = obj75 ']' 2024-04-25T09:18:09.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj53 = obj75 ']' 2024-04-25T09:18:09.346 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj54 = obj75 ']' 2024-04-25T09:18:09.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj55 = obj75 ']' 2024-04-25T09:18:09.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj56 = obj75 ']' 2024-04-25T09:18:09.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj57 = obj75 ']' 2024-04-25T09:18:09.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj58 = obj75 ']' 2024-04-25T09:18:09.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj59 = obj75 ']' 2024-04-25T09:18:09.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj60 = obj75 ']' 2024-04-25T09:18:09.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj61 = obj75 ']' 2024-04-25T09:18:09.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj62 = obj75 ']' 2024-04-25T09:18:09.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj63 = obj75 ']' 2024-04-25T09:18:09.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj64 = obj75 ']' 2024-04-25T09:18:09.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj65 = obj75 ']' 2024-04-25T09:18:09.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.732 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj66 = obj75 ']' 2024-04-25T09:18:09.733 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj67 = obj75 ']' 2024-04-25T09:18:09.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj68 = obj75 ']' 2024-04-25T09:18:09.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj69 = obj75 ']' 2024-04-25T09:18:09.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj70 = obj75 ']' 2024-04-25T09:18:09.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj71 = obj75 ']' 2024-04-25T09:18:09.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj72 = obj75 ']' 2024-04-25T09:18:09.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj73 = obj75 ']' 2024-04-25T09:18:09.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:09.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:09.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj74 = obj75 ']' 2024-04-25T09:18:09.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj75 = obj75 ']' 2024-04-25T09:18:10.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:406: TEST_rep_recovery_unfound: rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.044 INFO:tasks.workunit.client.0.smithi102.stderr:error getting test-pool/obj75: (2) No such file or directory 2024-04-25T09:18:10.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj76 = obj75 ']' 2024-04-25T09:18:10.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj77 = obj75 ']' 2024-04-25T09:18:10.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj78 = obj75 ']' 2024-04-25T09:18:10.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj79 = obj75 ']' 2024-04-25T09:18:10.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj80 = obj75 ']' 2024-04-25T09:18:10.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj81 = obj75 ']' 2024-04-25T09:18:10.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj82 = obj75 ']' 2024-04-25T09:18:10.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.253 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj83 = obj75 ']' 2024-04-25T09:18:10.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj84 = obj75 ']' 2024-04-25T09:18:10.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj85 = obj75 ']' 2024-04-25T09:18:10.314 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj86 = obj75 ']' 2024-04-25T09:18:10.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj87 = obj75 ']' 2024-04-25T09:18:10.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj88 = obj75 ']' 2024-04-25T09:18:10.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj89 = obj75 ']' 2024-04-25T09:18:10.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj90 = obj75 ']' 2024-04-25T09:18:10.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj91 = obj75 ']' 2024-04-25T09:18:10.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.541 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj92 = obj75 ']' 2024-04-25T09:18:10.542 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj93 = obj75 ']' 2024-04-25T09:18:10.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj94 = obj75 ']' 2024-04-25T09:18:10.618 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj95 = obj75 ']' 2024-04-25T09:18:10.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj96 = obj75 ']' 2024-04-25T09:18:10.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj97 = obj75 ']' 2024-04-25T09:18:10.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj98 = obj75 ']' 2024-04-25T09:18:10.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj99 = obj75 ']' 2024-04-25T09:18:10.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:18:10.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj100 = obj75 ']' 2024-04-25T09:18:10.797 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:413: TEST_rep_recovery_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:18:10.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:415: TEST_rep_recovery_unfound: delete_pool test-pool 2024-04-25T09:18:10.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-04-25T09:18:10.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-04-25T09:18:11.308 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test-pool' does not exist 2024-04-25T09:18:11.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:18:11.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:11.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:18:11.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:18:11.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:18:11.325 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:18:11.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:18:11.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:18:11.325 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:18:11.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:18:11.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:18:11.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:18:11.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:18:11.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:18:11.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:18:11.436 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:18:11.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:11.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:18:11.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:18:11.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:11.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:18:11.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:18:11.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:18:11.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:18:11.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:11.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:18:11.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.388535 2024-04-25T09:18:11.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:18:11.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:18:11.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:18:11.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:11.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:11.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:18:11.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:18:11.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:18:11.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:18:11.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:18:11.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:18:11.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:18:11.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:18:11.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:18:11.468 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:18:11.468 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:18:11.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:18:11.469 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:18:11.469 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:18:11.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:11.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:18:11.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:18:11.471 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:11.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:18:11.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:18:11.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:18:11.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:18:11.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:11.473 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.388535 2024-04-25T09:18:11.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.388535 2024-04-25T09:18:11.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:18:11.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:18:11.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:18:11.475 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:18:11.476 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:18:11.528 INFO:tasks.workunit:Running workunit osd/osd-reuse-id.sh... 2024-04-25T09:18:11.528 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/osd-reuse-id.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh 2024-04-25T09:18:11.581 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-reuse-id 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:20: run: local dir=td/osd-reuse-id 2024-04-25T09:18:11.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:21: run: shift 2024-04-25T09:18:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: export CEPH_MON=127.0.0.1:7123 2024-04-25T09:18:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: CEPH_MON=127.0.0.1:7123 2024-04-25T09:18:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:24: run: export CEPH_ARGS 2024-04-25T09:18:11.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: uuidgen 2024-04-25T09:18:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: CEPH_ARGS+='--fsid=78642707-40ea-4122-a887-b8ddc6e51d59 --auth-supported=none ' 2024-04-25T09:18:11.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7123 ' 2024-04-25T09:18:11.587 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: set 2024-04-25T09:18:11.588 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:18:11.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: local funcs=TEST_reuse_id 2024-04-25T09:18:11.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:29: run: for func in $funcs 2024-04-25T09:18:11.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:30: run: setup td/osd-reuse-id 2024-04-25T09:18:11.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reuse-id 2024-04-25T09:18:11.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reuse-id 2024-04-25T09:18:11.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:18:11.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:18:11.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:18:11.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:18:11.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:18:11.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:18:11.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:18:11.590 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:18:11.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:18:11.591 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:18:11.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:18:11.592 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:18:11.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:18:11.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:18:11.593 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:18:11.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:11.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:18:11.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:18:11.595 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:11.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:18:11.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:18:11.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:18:11.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:18:11.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:11.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:18:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.444699 2024-04-25T09:18:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:18:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:18:11.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reuse-id 2024-04-25T09:18:11.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:18:11.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:11.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:18:11.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.444699 2024-04-25T09:18:11.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reuse-id 1' TERM HUP INT 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:31: run: TEST_reuse_id td/osd-reuse-id 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:37: TEST_reuse_id: local dir=td/osd-reuse-id 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:39: TEST_reuse_id: run_mon td/osd-reuse-id a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reuse-id 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reuse-id/a 2024-04-25T09:18:11.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reuse-id/a --run-dir=td/osd-reuse-id --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:18:12.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:18:12.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:12.135 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:12.136 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:12.136 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:12.136 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:18:12.136 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444699/$cluster-$name.asok' 2024-04-25T09:18:12.136 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reuse-id/a '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.444699/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reuse-id/log --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:18:12.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:18:12.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:18:12.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:18:12.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:18:12.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:18:12.353 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:18:12.353 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:18:12.353 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:18:12.353 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:18:12.353 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:12.353 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:12.353 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:18:12.353 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.444699/ceph-mon.a.asok 2024-04-25T09:18:12.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:18:12.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.444699/ceph-mon.a.asok config get fsid 2024-04-25T09:18:12.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:18:12.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:18:12.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:18:12.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:18:12.466 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:18:12.466 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:18:12.466 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:18:12.466 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:18:12.467 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:12.467 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:12.467 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:18:12.467 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.444699/ceph-mon.a.asok 2024-04-25T09:18:12.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:18:12.467 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.444699/ceph-mon.a.asok config get mon_host 2024-04-25T09:18:12.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:40: TEST_reuse_id: run_mgr td/osd-reuse-id x 2024-04-25T09:18:12.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reuse-id 2024-04-25T09:18:12.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:18:12.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:18:12.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:18:12.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reuse-id/x 2024-04-25T09:18:12.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:18:13.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:18:13.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:13.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:13.488 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:13.488 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:13.488 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:18:13.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444699/$cluster-$name.asok' 2024-04-25T09:18:13.489 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:18:13.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reuse-id/x '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.444699/$cluster-$name.asok' --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:18:13.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:41: TEST_reuse_id: run_osd td/osd-reuse-id 0 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/0 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=78642707-40ea-4122-a887-b8ddc6e51d59 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/0' 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/0/journal' 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:18:13.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444699/$cluster-$name.asok' 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444699/$cluster-$name.asok' 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:18:13.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:18:13.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:18:13.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:18:13.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:18:13.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:18:13.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:18:13.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:18:13.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:18:13.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/0 2024-04-25T09:18:13.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:18:13.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0d562a24-a172-4419-9941-dde155c39c26 2024-04-25T09:18:13.561 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 0d562a24-a172-4419-9941-dde155c39c26 2024-04-25T09:18:13.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0d562a24-a172-4419-9941-dde155c39c26' 2024-04-25T09:18:13.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:18:13.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVHypmL35LIxAAaf7DiONkMyz7HOZZcENhrg== 2024-04-25T09:18:13.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVHypmL35LIxAAaf7DiONkMyz7HOZZcENhrg=="}' 2024-04-25T09:18:13.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0d562a24-a172-4419-9941-dde155c39c26 -i td/osd-reuse-id/0/new.json 2024-04-25T09:18:13.886 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:18:13.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/0/new.json 2024-04-25T09:18:13.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=78642707-40ea-4122-a887-b8ddc6e51d59 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.444699/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVHypmL35LIxAAaf7DiONkMyz7HOZZcENhrg== --osd-uuid 0d562a24-a172-4419-9941-dde155c39c26 2024-04-25T09:18:13.933 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:13.947+0000 7f32995127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:13.963 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:13.975+0000 7f32995127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:13.965 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:13.979+0000 7f32995127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:13.966 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:13.979+0000 7f32995127c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:18:13.966 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:13.979+0000 7f32995127c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:18:13.966 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:13.979+0000 7f32995127c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:18:13.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:14.007+0000 7f32995127c0 -1 bluestore(td/osd-reuse-id/0) _read_fsid unparsable uuid 2024-04-25T09:18:20.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/0/keyring 2024-04-25T09:18:20.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:18:20.429 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:18:20.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:18:20.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:18:20.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:18:20.965 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:18:20.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:18:20.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:18:20.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:18:20.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=78642707-40ea-4122-a887-b8ddc6e51d59 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.444699/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:18:20.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:21.006+0000 7fb052c457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:21.037 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:21.050+0000 7fb052c457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:21.038 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:21.050+0000 7fb052c457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:21.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:18:21.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:18:21.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:18:21.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:18:21.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:18:21.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:21.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:18:21.401 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:18:21.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:21.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:21.625 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:21.638+0000 7fb052c457c0 -1 Falling back to public interface 2024-04-25T09:18:21.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:22.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:22.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:22.837 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:18:22.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:18:22.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:22.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:23.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:24.093 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:24.106+0000 7fb052c457c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:18:24.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:24.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:24.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:18:24.250 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:18:24.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:24.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:24.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:25.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:25.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:25.794 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:18:25.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:18:25.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:25.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:26.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:27.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:27.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:27.247 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:18:27.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:18:27.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:27.248 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:27.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:28.708 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:28.708 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:18:28.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:18:28.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:28.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:29.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:30.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:30.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:30.203 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:18:30.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:18:30.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:30.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:30.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:31.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:31.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:31.626 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:18:31.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:18:31.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:31.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:32.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:33.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:33.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:33.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:18:33.057 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:18:33.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:33.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:33.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:18:34.527 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:18:34.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:34.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:35.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:36.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:36.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:36.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:18:36.115 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:18:36.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:36.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:36.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:37.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:37.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:37.584 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:18:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:18:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:37.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:38.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:39.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:39.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:39.491 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:18:39.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:18:39.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:39.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:40.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:41.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:41.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:41.539 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:18:41.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:18:41.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:41.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:42.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:43.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:43.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:43.456 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:18:43.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:18:43.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:43.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:44.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:45.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:45.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:45.508 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:18:45.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:18:45.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:45.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:46.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:47.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:47.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:47.239 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:18:47.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:18:47.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:47.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:48.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:49.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:49.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:49.076 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:18:49.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:18:49.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:49.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:49.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:50.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:50.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:50.557 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:18:50.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:18:50.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:50.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:51.018 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1316756785,v1:127.0.0.1:6803/1316756785] [v2:127.0.0.1:6804/1316756785,v1:127.0.0.1:6805/1316756785] exists,up 0d562a24-a172-4419-9941-dde155c39c26 2024-04-25T09:18:51.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:18:51.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:18:51.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:18:51.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:42: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-04-25T09:18:51.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:18:51.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:18:51.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=78642707-40ea-4122-a887-b8ddc6e51d59 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:18:51.022 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444699/$cluster-$name.asok' 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444699/$cluster-$name.asok' 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:18:51.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:18:51.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:18:51.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:18:51.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:18:51.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:18:51.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:18:51.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-04-25T09:18:51.024 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:18:51.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b02b64e3-4a98-403d-9730-cd306b0081db 2024-04-25T09:18:51.025 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 b02b64e3-4a98-403d-9730-cd306b0081db 2024-04-25T09:18:51.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b02b64e3-4a98-403d-9730-cd306b0081db' 2024-04-25T09:18:51.025 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:18:51.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7HypmsoJqAxAA0uxVXS4VCtOfVjYx+MrP8g== 2024-04-25T09:18:51.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7HypmsoJqAxAA0uxVXS4VCtOfVjYx+MrP8g=="}' 2024-04-25T09:18:51.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b02b64e3-4a98-403d-9730-cd306b0081db -i td/osd-reuse-id/1/new.json 2024-04-25T09:18:51.528 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:18:51.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-04-25T09:18:51.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=78642707-40ea-4122-a887-b8ddc6e51d59 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.444699/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7HypmsoJqAxAA0uxVXS4VCtOfVjYx+MrP8g== --osd-uuid b02b64e3-4a98-403d-9730-cd306b0081db 2024-04-25T09:18:51.578 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:51.590+0000 7f7b6cf137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:51.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:51.634+0000 7f7b6cf137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:51.621 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:51.634+0000 7f7b6cf137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:51.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:51.634+0000 7f7b6cf137c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:18:51.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:51.634+0000 7f7b6cf137c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:18:51.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:51.634+0000 7f7b6cf137c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:18:51.659 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:51.670+0000 7f7b6cf137c0 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-04-25T09:18:58.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-04-25T09:18:58.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:18:58.590 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:18:58.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:18:58.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:18:59.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:18:59.161 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:18:59.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:18:59.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=78642707-40ea-4122-a887-b8ddc6e51d59 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.444699/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:18:59.161 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:18:59.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:18:59.190 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:59.202+0000 7f35b7e6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:59.193 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:59.206+0000 7f35b7e6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:59.195 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:59.206+0000 7f35b7e6b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:59.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:18:59.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:18:59.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:18:59.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:18:59.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:18:59.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:59.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:18:59.592 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:18:59.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:59.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:18:59.765 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:18:59.778+0000 7f35b7e6b7c0 -1 Falling back to public interface 2024-04-25T09:19:00.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:01.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:01.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:01.030 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:19:01.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:19:01.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:01.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:01.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:02.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:02.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:02.438 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:19:02.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:19:02.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:02.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:02.464 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:02.478+0000 7f35b7e6b7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:19:02.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:03.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:03.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:03.868 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:19:03.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:19:03.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:03.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:04.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:05.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:05.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:05.304 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:19:05.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:19:05.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:05.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:05.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:06.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:06.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:06.910 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:19:06.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:19:06.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:06.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:07.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:08.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:08.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:08.414 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:19:08.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:19:08.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:08.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:08.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:09.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:09.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:09.917 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:19:09.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:19:09.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:09.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:10.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:11.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:11.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:11.646 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:19:11.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:19:11.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:11.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:12.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:13.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:13.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:13.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:19:13.088 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:19:13.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:13.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:13.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:14.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:14.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:14.635 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:19:14.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:19:14.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:14.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:15.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:16.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:16.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:16.285 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:19:16.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:19:16.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:16.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:16.742 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:17.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:17.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:17.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:19:17.744 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:19:17.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:17.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:18.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:19.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:19.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:19.193 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:19:19.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:19:19.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:19.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:20.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:21.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:21.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:21.118 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:19:21.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:19:21.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:21.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:21.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:22.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:22.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:22.986 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:19:22.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:19:22.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:22.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:24.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:25.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:25.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:25.103 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:19:25.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:19:25.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:25.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:25.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:26.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:26.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:26.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:19:26.923 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:19:26.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:26.923 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:27.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:28.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:28.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:28.948 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:19:28.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:19:28.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:28.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:29.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:30.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:30.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:30.655 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T09:19:30.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:19:30.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:30.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:31.114 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3048717275,v1:127.0.0.1:6811/3048717275] [v2:127.0.0.1:6812/3048717275,v1:127.0.0.1:6813/3048717275] exists,up b02b64e3-4a98-403d-9730-cd306b0081db 2024-04-25T09:19:31.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:19:31.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:19:31.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:19:31.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:43: TEST_reuse_id: create_rbd_pool 2024-04-25T09:19:31.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T09:19:31.522 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' does not exist 2024-04-25T09:19:31.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T09:19:31.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T09:19:32.125 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'rbd' already exists 2024-04-25T09:19:32.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:19:33.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T09:19:34.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:44: TEST_reuse_id: wait_for_clean 2024-04-25T09:19:34.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:19:34.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:19:34.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:19:34.898 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:19:34.899 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:19:34.899 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:19:34.899 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:19:34.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:19:34.900 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:19:35.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:19:35.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:19:35.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:19:35.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:19:35.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:19:35.017 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:19:35.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:19:35.441 INFO:tasks.workunit.client.0.smithi102.stderr:1' 2024-04-25T09:19:35.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:19:35.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:19:35.442 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:19:35.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836495 2024-04-25T09:19:35.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836495 2024-04-25T09:19:35.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836495' 2024-04-25T09:19:35.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:19:35.640 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:19:35.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672965 2024-04-25T09:19:35.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672965 2024-04-25T09:19:35.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836495 1-42949672965' 2024-04-25T09:19:35.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:19:35.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836495 2024-04-25T09:19:35.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:19:35.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:19:35.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836495 2024-04-25T09:19:35.796 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:19:35.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836495 2024-04-25T09:19:35.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-04-25T09:19:35.797 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836495 2024-04-25T09:19:35.797 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:19:36.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836495 2024-04-25T09:19:36.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:19:37.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:19:37.192 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:19:37.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836495 -lt 21474836495 2024-04-25T09:19:37.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:19:37.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672965 2024-04-25T09:19:37.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:19:37.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:19:37.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672965 2024-04-25T09:19:37.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:19:37.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672965 2024-04-25T09:19:37.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-04-25T09:19:37.660 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672965 2024-04-25T09:19:37.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:19:38.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672965 -lt 42949672965 2024-04-25T09:19:38.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:19:38.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:19:38.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:19:38.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-25T09:19:38.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:19:38.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:19:38.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:19:38.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:19:38.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:19:38.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:19:38.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:19:39.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-25T09:19:39.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:19:39.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:19:39.118 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:19:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-25T09:19:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:19:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:19:39.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:45: TEST_reuse_id: destroy_osd td/osd-reuse-id 1 2024-04-25T09:19:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/osd-reuse-id 2024-04-25T09:19:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=1 2024-04-25T09:19:39.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.1 2024-04-25T09:19:40.150 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already out. 2024-04-25T09:19:40.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/osd-reuse-id TERM osd.1 2024-04-25T09:19:40.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:19:40.167 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:19:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:19:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:19:40.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:19:41.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:19:41.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.1 2024-04-25T09:19:41.899 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 is already down. 2024-04-25T09:19:41.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.1 --yes-i-really-mean-it 2024-04-25T09:19:42.418 INFO:tasks.workunit.client.0.smithi102.stderr:osd.1 does not exist 2024-04-25T09:19:42.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/osd-reuse-id/1 2024-04-25T09:19:42.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id/1 2024-04-25T09:19:42.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:19:42.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id/1 KILL 2024-04-25T09:19:42.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:19:42.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:19:42.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:19:42.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:19:42.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:19:42.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:19:42.438 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:19:42.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:19:42.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:19:42.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:19:42.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:19:42.440 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:19:42.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:42.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:19:42.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:19:42.441 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:42.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:19:42.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:19:42.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id/1 2024-04-25T09:19:42.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:19:42.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:42.447 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:19:42.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.444699 2024-04-25T09:19:42.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:19:42.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:19:42.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/osd-reuse-id/1 2024-04-25T09:19:42.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:46: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-04-25T09:19:42.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:19:42.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:19:42.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:19:42.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:19:42.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-04-25T09:19:42.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=78642707-40ea-4122-a887-b8ddc6e51d59 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:19:42.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.444699/$cluster-$name.asok' 2024-04-25T09:19:42.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.444699/$cluster-$name.asok' 2024-04-25T09:19:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:19:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:19:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:19:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:19:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:19:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:19:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:19:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:19:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:19:42.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:19:42.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:19:42.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-04-25T09:19:42.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:19:42.454 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 b1c9e973-f917-46bf-bba8-f630e1146ef5 2024-04-25T09:19:42.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b1c9e973-f917-46bf-bba8-f630e1146ef5 2024-04-25T09:19:42.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b1c9e973-f917-46bf-bba8-f630e1146ef5' 2024-04-25T09:19:42.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:19:42.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuICpmtz0WHRAA2t8O63ODVb0hRF57OaFScQ== 2024-04-25T09:19:42.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuICpmtz0WHRAA2t8O63ODVb0hRF57OaFScQ=="}' 2024-04-25T09:19:42.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b1c9e973-f917-46bf-bba8-f630e1146ef5 -i td/osd-reuse-id/1/new.json 2024-04-25T09:19:43.018 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:19:43.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-04-25T09:19:43.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=78642707-40ea-4122-a887-b8ddc6e51d59 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.444699/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAuICpmtz0WHRAA2t8O63ODVb0hRF57OaFScQ== --osd-uuid b1c9e973-f917-46bf-bba8-f630e1146ef5 2024-04-25T09:19:43.066 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:43.077+0000 7f4daeb5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:43.111 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:43.125+0000 7f4daeb5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:43.113 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:43.125+0000 7f4daeb5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:43.113 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:43.125+0000 7f4daeb5e7c0 -1 asok(0x5557c3388000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.444699/ceph-osd.1.asok': (2) No such file or directory 2024-04-25T09:19:43.113 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:43.125+0000 7f4daeb5e7c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:19:43.114 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:43.125+0000 7f4daeb5e7c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:19:43.114 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:43.125+0000 7f4daeb5e7c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:19:43.158 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:43.169+0000 7f4daeb5e7c0 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-04-25T09:19:49.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-04-25T09:19:49.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:19:49.739 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:19:49.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:19:49.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:19:50.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:19:50.307 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:19:50.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:19:50.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=78642707-40ea-4122-a887-b8ddc6e51d59 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.444699/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:19:50.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:19:50.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:19:50.335 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:50.349+0000 7f0e5fc977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:50.338 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:50.349+0000 7f0e5fc977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:50.353+0000 7f0e5fc977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:50.339 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:50.353+0000 7f0e5fc977c0 -1 asok(0x55f4d5f40000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.444699/ceph-osd.1.asok': (2) No such file or directory 2024-04-25T09:19:50.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:19:50.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:19:50.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:19:50.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:19:50.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:19:50.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:50.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:19:50.700 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:19:50.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:50.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:50.908 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:50.921+0000 7f0e5fc977c0 -1 Falling back to public interface 2024-04-25T09:19:51.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:52.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:52.146 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:19:52.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:19:52.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:52.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:52.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:53.544 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:53.557+0000 7f0e5fc977c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:19:53.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:53.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:53.555 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:19:53.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:19:53.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:53.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:54.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:55.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:55.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:55.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:19:55.077 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:19:55.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:55.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:55.502 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1741318697,v1:127.0.0.1:6811/1741318697] [v2:127.0.0.1:6812/1741318697,v1:127.0.0.1:6813/1741318697] exists,up b1c9e973-f917-46bf-bba8-f630e1146ef5 2024-04-25T09:19:55.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:19:55.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:19:55.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:19:55.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:32: run: teardown td/osd-reuse-id 2024-04-25T09:19:55.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:19:55.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:19:55.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:19:55.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:19:55.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:19:55.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:19:55.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:19:55.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:19:55.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:19:55.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:19:55.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:19:55.613 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:19:55.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:19:55.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:19:55.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:19:55.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:55.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:19:55.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:19:55.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:55.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:19:55.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:19:55.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:19:55.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:19:55.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:55.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:19:55.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.444699 2024-04-25T09:19:55.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:19:55.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:19:55.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:19:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-reuse-id 0 2024-04-25T09:19:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:19:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:19:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:19:55.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:19:55.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:19:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:19:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:19:55.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:19:55.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:19:55.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:19:55.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:19:55.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:19:55.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:19:55.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:19:55.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:19:55.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:55.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:19:55.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:19:55.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:55.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:19:55.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:19:55.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:19:55.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:19:55.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:55.636 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.444699 2024-04-25T09:19:55.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.444699 2024-04-25T09:19:55.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:19:55.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:19:55.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:19:55.638 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:19:55.638 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:19:55.688 INFO:tasks.workunit:Running workunit osd/pg-split-merge.sh... 2024-04-25T09:19:55.688 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/pg-split-merge.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh 2024-04-25T09:19:55.740 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:19:55.746 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/pg-split-merge 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:5: run: local dir=td/pg-split-merge 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:6: run: shift 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: export CEPH_MON=127.0.0.1:7147 2024-04-25T09:19:55.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: CEPH_MON=127.0.0.1:7147 2024-04-25T09:19:55.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:9: run: export CEPH_ARGS 2024-04-25T09:19:55.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: uuidgen 2024-04-25T09:19:55.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: CEPH_ARGS+='--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none ' 2024-04-25T09:19:55.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:19:55.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: set 2024-04-25T09:19:55.748 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:19:55.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: local 'funcs=TEST_a_merge_empty 2024-04-25T09:19:55.749 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_import_after_merge_and_gap 2024-04-25T09:19:55.749 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_import_after_split' 2024-04-25T09:19:55.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:19:55.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:19:55.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:19:55.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:19:55.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:19:55.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:19:55.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:19:55.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:19:55.750 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:19:55.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:19:55.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:19:55.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:19:55.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:19:55.751 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:19:55.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:19:55.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:19:55.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:19:55.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:19:55.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:19:55.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:55.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:19:55.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:19:55.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:19:55.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:19:55.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:19:55.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:19:55.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:19:55.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:55.757 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:19:55.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.450371 2024-04-25T09:19:55.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:19:55.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:19:55.757 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:19:55.758 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:19:55.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:55.759 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:19:55.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.450371 2024-04-25T09:19:55.761 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:19:55.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:19:55.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:19:55.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:19:55.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:19:55.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_a_merge_empty td/pg-split-merge 2024-04-25T09:19:55.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:22: TEST_a_merge_empty: local dir=td/pg-split-merge 2024-04-25T09:19:55.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:24: TEST_a_merge_empty: run_mon td/pg-split-merge a --osd_pool_default_size=3 2024-04-25T09:19:55.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:19:55.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:19:55.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:19:55.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:19:55.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:19:55.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=3 2024-04-25T09:19:56.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:19:56.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:19:56.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:19:56.247 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:19:56.247 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:56.247 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:19:56.247 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:19:56.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T09:19:56.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:19:56.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:19:56.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:19:56.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:19:56.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:19:56.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:19:56.505 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:19:56.505 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:19:56.505 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:19:56.505 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:19:56.505 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:56.505 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:19:56.505 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.450371/ceph-mon.a.asok 2024-04-25T09:19:56.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:19:56.506 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.450371/ceph-mon.a.asok config get fsid 2024-04-25T09:19:56.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:19:56.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:19:56.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:19:56.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:19:56.612 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:19:56.613 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:19:56.613 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:19:56.613 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:19:56.613 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:19:56.613 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:56.613 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:19:56.613 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.450371/ceph-mon.a.asok 2024-04-25T09:19:56.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:19:56.614 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.450371/ceph-mon.a.asok config get mon_host 2024-04-25T09:19:56.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:25: TEST_a_merge_empty: run_mgr td/pg-split-merge x 2024-04-25T09:19:56.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:19:56.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:19:56.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:19:56.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:19:56.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:19:56.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:19:57.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:19:57.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:19:57.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:19:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:19:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:57.629 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:19:57.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:19:57.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:19:57.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:19:57.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:26: TEST_a_merge_empty: run_osd td/pg-split-merge 0 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:19:57.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:19:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:19:57.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:19:57.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:19:57.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:19:57.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:19:57.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:19:57.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:19:57.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2b715d81-62ed-4948-974e-debe198a9658 2024-04-25T09:19:57.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2b715d81-62ed-4948-974e-debe198a9658' 2024-04-25T09:19:57.671 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 2b715d81-62ed-4948-974e-debe198a9658 2024-04-25T09:19:57.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:19:57.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9ICpmnwHkKRAAUyFVnQRH7cJazYW5rT1ybg== 2024-04-25T09:19:57.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9ICpmnwHkKRAAUyFVnQRH7cJazYW5rT1ybg=="}' 2024-04-25T09:19:57.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2b715d81-62ed-4948-974e-debe198a9658 -i td/pg-split-merge/0/new.json 2024-04-25T09:19:57.994 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:19:58.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:19:58.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9ICpmnwHkKRAAUyFVnQRH7cJazYW5rT1ybg== --osd-uuid 2b715d81-62ed-4948-974e-debe198a9658 2024-04-25T09:19:58.040 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:58.053+0000 7fcd94d157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:58.071 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:58.085+0000 7fcd94d157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:58.073 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:58.085+0000 7fcd94d157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:58.074 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:58.085+0000 7fcd94d157c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:19:58.074 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:58.085+0000 7fcd94d157c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:19:58.074 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:58.085+0000 7fcd94d157c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:19:58.103 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:19:58.117+0000 7fcd94d157c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:20:04.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:20:04.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:20:04.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:20:04.646 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:20:04.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:20:05.213 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:20:05.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:20:05.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:20:05.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:20:05.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:20:05.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:20:05.247 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:05.256+0000 7fe1971087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:05.269 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:05.280+0000 7fe1971087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:05.270 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:05.280+0000 7fe1971087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:05.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:20:05.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:20:05.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:20:05.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:20:05.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:20:05.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:05.634 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:20:05.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:20:05.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:05.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:05.838 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:05.848+0000 7fe1971087c0 -1 Falling back to public interface 2024-04-25T09:20:06.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:07.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:07.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:07.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:20:07.035 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:20:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:07.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:07.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:08.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:08.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:08.444 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:20:08.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:20:08.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:08.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:08.493 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:08.504+0000 7fe1971087c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:20:09.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:10.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:10.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:10.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:20:10.024 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:20:10.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:10.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:10.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:11.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:11.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:11.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:20:11.518 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:20:11.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:11.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:11.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:12.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:12.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:12.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:20:12.972 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:20:12.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:12.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:13.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:14.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:14.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:14.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:20:14.572 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:20:14.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:14.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:15.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:16.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:16.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:16.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:20:16.037 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:20:16.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:16.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:16.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:17.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:17.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:17.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:20:17.490 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:20:17.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:17.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:17.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:18.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:18.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:18.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:20:18.956 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:20:18.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:18.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:19.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:20.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:20.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:20.454 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:20:20.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:20:20.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:20.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:20.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:21.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:21.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:21.906 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:20:21.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:20:21.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:21.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:22.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:23.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:23.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:23.863 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:20:23.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:20:23.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:23.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:24.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:25.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:25.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:25.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:20:25.777 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:20:25.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:25.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:26.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:27.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:27.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:27.795 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:20:27.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:20:27.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:27.796 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:28.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:29.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:29.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:29.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:20:29.642 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:20:29.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:29.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:30.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:31.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:31.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:31.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:20:31.577 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:20:31.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:31.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:32.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:33.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:33.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:33.277 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:20:33.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:20:33.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:33.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:34.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:35.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:35.039 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:20:35.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:20:35.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:35.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:20:35.457 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2265093928,v1:127.0.0.1:6803/2265093928] [v2:127.0.0.1:6804/2265093928,v1:127.0.0.1:6805/2265093928] exists,up 2b715d81-62ed-4948-974e-debe198a9658 2024-04-25T09:20:35.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:20:35.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:20:35.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:20:35.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:27: TEST_a_merge_empty: run_osd td/pg-split-merge 1 2024-04-25T09:20:35.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:20:35.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:20:35.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:20:35.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/1 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/1' 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/1/journal' 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:20:35.462 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:20:35.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/1 2024-04-25T09:20:35.464 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:20:35.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=46342f7a-c8d0-4aa3-ade1-8d10e81d63ee 2024-04-25T09:20:35.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 46342f7a-c8d0-4aa3-ade1-8d10e81d63ee' 2024-04-25T09:20:35.465 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 46342f7a-c8d0-4aa3-ade1-8d10e81d63ee 2024-04-25T09:20:35.465 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:20:35.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjICpmmYHnHRAAVByBGtYaIJw57xU6Yq6d5A== 2024-04-25T09:20:35.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjICpmmYHnHRAAVByBGtYaIJw57xU6Yq6d5A=="}' 2024-04-25T09:20:35.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 46342f7a-c8d0-4aa3-ade1-8d10e81d63ee -i td/pg-split-merge/1/new.json 2024-04-25T09:20:35.976 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:20:35.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/1/new.json 2024-04-25T09:20:35.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjICpmmYHnHRAAVByBGtYaIJw57xU6Yq6d5A== --osd-uuid 46342f7a-c8d0-4aa3-ade1-8d10e81d63ee 2024-04-25T09:20:36.030 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:36.040+0000 7f43e925c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:36.045 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:36.056+0000 7f43e925c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:36.046 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:36.056+0000 7f43e925c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:36.047 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:36.060+0000 7f43e925c7c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:20:36.047 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:36.060+0000 7f43e925c7c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:20:36.047 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:36.060+0000 7f43e925c7c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:20:36.084 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:36.096+0000 7f43e925c7c0 -1 bluestore(td/pg-split-merge/1) _read_fsid unparsable uuid 2024-04-25T09:20:43.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/1/keyring 2024-04-25T09:20:43.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:20:43.077 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:20:43.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:20:43.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:20:43.644 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:20:43.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:20:43.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:20:43.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:20:43.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:20:43.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:20:43.678 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:43.688+0000 7f4d74eb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:43.709 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:43.720+0000 7f4d74eb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:43.710 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:43.720+0000 7f4d74eb07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:44.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:20:44.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:20:44.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:20:44.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:20:44.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:20:44.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:44.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:20:44.100 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:20:44.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:44.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:44.285 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:44.296+0000 7f4d74eb07c0 -1 Falling back to public interface 2024-04-25T09:20:44.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:45.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:45.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:45.559 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:20:45.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:20:45.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:45.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:45.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:46.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:46.989 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:20:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:20:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:46.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:46.998 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:20:47.008+0000 7f4d74eb07c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:20:47.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:48.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:48.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:48.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:20:48.611 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:20:48.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:48.612 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:49.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:50.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:50.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:50.171 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:20:50.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:20:50.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:50.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:50.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:51.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:20:51.592 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:20:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:51.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:52.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:53.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:53.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:53.294 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:20:53.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:20:53.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:53.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:53.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:54.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:54.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:54.745 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:20:54.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:20:54.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:54.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:55.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:56.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:56.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:56.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:20:56.305 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:20:56.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:56.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:56.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:57.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:57.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:57.722 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:20:57.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:20:57.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:57.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:58.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:59.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:59.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:59.332 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:20:59.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:20:59.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:59.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:59.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:00.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:00.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:00.769 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:21:00.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:21:00.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:00.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:01.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:02.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:02.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:02.425 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:21:02.425 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:21:02.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:02.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:03.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:04.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:04.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:04.355 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:21:04.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:21:04.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:04.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:05.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:06.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:06.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:06.643 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:21:06.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:21:06.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:06.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:07.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:08.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:08.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:08.519 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:21:08.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:21:08.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:08.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:09.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:10.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:10.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:10.223 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:21:10.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:21:10.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:10.224 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:11.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:12.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:12.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:12.042 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:21:12.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:21:12.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:12.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:13.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:14.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:14.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:14.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:21:14.079 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:21:14.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:14.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:21:14.515 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/781881549,v1:127.0.0.1:6811/781881549] [v2:127.0.0.1:6812/781881549,v1:127.0.0.1:6813/781881549] exists,up 46342f7a-c8d0-4aa3-ade1-8d10e81d63ee 2024-04-25T09:21:14.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:21:14.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:21:14.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:28: TEST_a_merge_empty: run_osd td/pg-split-merge 2 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/2 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:21:14.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:21:14.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:21:14.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:21:14.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:21:14.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:21:14.520 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:21:14.520 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:14.520 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/2 2024-04-25T09:21:14.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:21:14.522 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 c306efe5-4b15-4852-baf6-93fe59da434b 2024-04-25T09:21:14.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c306efe5-4b15-4852-baf6-93fe59da434b 2024-04-25T09:21:14.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c306efe5-4b15-4852-baf6-93fe59da434b' 2024-04-25T09:21:14.522 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:21:14.544 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCKICpmQD9oIRAAnFcrti2hlttumcOUE3eBFQ== 2024-04-25T09:21:14.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCKICpmQD9oIRAAnFcrti2hlttumcOUE3eBFQ=="}' 2024-04-25T09:21:14.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c306efe5-4b15-4852-baf6-93fe59da434b -i td/pg-split-merge/2/new.json 2024-04-25T09:21:15.039 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:21:15.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/2/new.json 2024-04-25T09:21:15.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKICpmQD9oIRAAnFcrti2hlttumcOUE3eBFQ== --osd-uuid c306efe5-4b15-4852-baf6-93fe59da434b 2024-04-25T09:21:15.086 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:15.099+0000 7f19230d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:15.111 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:15.127+0000 7f19230d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:15.112 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:15.127+0000 7f19230d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:15.113 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:15.127+0000 7f19230d97c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:21:15.113 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:15.127+0000 7f19230d97c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:21:15.113 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:15.127+0000 7f19230d97c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:21:15.150 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:15.163+0000 7f19230d97c0 -1 bluestore(td/pg-split-merge/2) _read_fsid unparsable uuid 2024-04-25T09:21:21.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/2/keyring 2024-04-25T09:21:21.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:21:21.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:21:21.869 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T09:21:21.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:21:22.428 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:21:22.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:21:22.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:21:22.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:21:22.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:21:22.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:21:22.455 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:22.471+0000 7f34bb46c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:22.458 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:22.471+0000 7f34bb46c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:22.459 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:22.475+0000 7f34bb46c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:22.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:21:22.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:21:22.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:21:22.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:21:22.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:21:22.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:22.856 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:21:22.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:21:22.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:22.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:23.029 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:23.043+0000 7f34bb46c7c0 -1 Falling back to public interface 2024-04-25T09:21:23.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:24.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:24.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:24.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:21:24.349 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:21:24.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:24.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:25.707 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:21:25.723+0000 7f34bb46c7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:21:25.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:25.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:25.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:21:25.783 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:21:25.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:25.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:26.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:27.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:27.560 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:21:27.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:27.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:21:27.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:27.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:28.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:29.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:29.024 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:21:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:21:29.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:29.025 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:29.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:30.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:30.566 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:21:30.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:30.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:21:30.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:30.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:31.167 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:32.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:32.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:32.170 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:21:32.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:21:32.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:32.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:32.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:33.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:33.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:33.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:21:33.792 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:21:33.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:33.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:34.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:35.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:35.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:35.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:21:35.257 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:21:35.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:35.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:35.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:36.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:36.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:36.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:21:36.791 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:21:36.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:36.791 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:37.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:38.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:38.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:38.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:21:38.305 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:21:38.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:38.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:38.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:39.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:39.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:39.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:21:39.773 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:21:39.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:39.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:40.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:41.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:41.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:41.298 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:21:41.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:21:41.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:41.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:42.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:43.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:43.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:43.561 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:21:43.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:21:43.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:43.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:44.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:45.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:45.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:45.574 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:21:45.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:21:45.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:45.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:46.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:47.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:47.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:47.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:21:47.697 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:21:47.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:47.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:48.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:49.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:49.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:49.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:21:49.416 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:21:49.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:49.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:50.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:51.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:51.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:51.242 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:21:51.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:21:51.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:51.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:51.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:52.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:52.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:52.872 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:21:52.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:21:52.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:52.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:21:53.318 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2749011928,v1:127.0.0.1:6819/2749011928] [v2:127.0.0.1:6820/2749011928,v1:127.0.0.1:6821/2749011928] exists,up c306efe5-4b15-4852-baf6-93fe59da434b 2024-04-25T09:21:53.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:21:53.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:21:53.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:21:53.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:30: TEST_a_merge_empty: ceph osd pool create foo 2 2024-04-25T09:21:53.909 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T09:21:53.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:31: TEST_a_merge_empty: ceph osd pool set foo pgp_num 1 2024-04-25T09:21:55.625 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 pgp_num to 1 2024-04-25T09:21:55.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:33: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:21:55.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:21:55.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:21:55.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:21:55.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:21:55.648 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:21:55.648 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:21:55.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:21:55.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:21:55.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:21:55.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:21:55.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:21:55.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:21:55.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:21:55.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:21:55.776 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:21:56.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:21:56.357 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:21:56.357 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:21:56.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:21:56.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:21:56.357 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:21:56.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836508 2024-04-25T09:21:56.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836508 2024-04-25T09:21:56.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508' 2024-04-25T09:21:56.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:21:56.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:21:56.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672975 2024-04-25T09:21:56.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672975 2024-04-25T09:21:56.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672975' 2024-04-25T09:21:56.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:21:56.779 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:21:56.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509449 2024-04-25T09:21:56.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509449 2024-04-25T09:21:56.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672975 2-64424509449' 2024-04-25T09:21:56.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:21:56.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836508 2024-04-25T09:21:56.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:21:56.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:21:56.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-04-25T09:21:56.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:21:56.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836508 2024-04-25T09:21:56.954 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836508 2024-04-25T09:21:56.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-04-25T09:21:56.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:21:57.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836508 2024-04-25T09:21:57.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:21:58.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:21:58.359 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:21:58.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836508 2024-04-25T09:21:58.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:21:58.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672975 2024-04-25T09:21:58.837 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:21:58.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:21:58.839 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672975 2024-04-25T09:21:58.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:21:58.841 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672975 2024-04-25T09:21:58.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672975 2024-04-25T09:21:58.841 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-04-25T09:21:58.841 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:21:59.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672975 -lt 42949672975 2024-04-25T09:21:59.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:21:59.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509449 2024-04-25T09:21:59.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:21:59.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:21:59.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509449 2024-04-25T09:21:59.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:21:59.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509449 2024-04-25T09:21:59.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-04-25T09:21:59.310 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509449 2024-04-25T09:21:59.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:21:59.744 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509449 2024-04-25T09:21:59.744 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:21:59.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:21:59.745 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:00.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:22:00.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:00.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:00.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:00.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:00.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:00.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:00.385 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:00.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:22:00.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:00.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:00.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:01.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T09:22:01.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' -1 2024-04-25T09:22:01.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:22:01.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-04-25T09:22:01.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:22:01.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:22:01.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:22:01.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:01.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:01.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:01.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:01.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:01.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:01.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:01.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:22:01.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:01.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:01.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:02.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T09:22:02.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T09:22:02.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:22:02.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:22:02.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:22:02.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:22:02.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:22:02.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:22:02.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:22:02.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:22:02.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:22:02.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T09:22:02.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:22:02.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T09:22:03.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:22:03.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:03.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:03.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:03.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:03.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:03.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:03.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:03.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:22:03.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:03.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:03.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T09:22:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T09:22:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:22:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:22:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:22:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:22:04.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:22:04.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:22:04.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:22:04.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:22:04.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:22:04.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-25T09:22:04.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:22:04.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T09:22:05.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:22:05.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:05.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:05.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:05.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:05.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:05.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:05.139 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:05.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:22:05.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:05.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:05.547 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T09:22:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T09:22:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:22:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:22:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:22:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:22:06.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:22:06.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:22:06.078 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:22:06.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:22:06.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:22:06.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-04-25T09:22:06.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:22:06.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T09:22:07.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:22:07.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:07.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:07.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:07.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:07.410 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:07.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:07.411 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:07.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:22:07.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:07.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:07.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:08.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:22:08.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:22:08.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:22:08.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:37: TEST_a_merge_empty: rados -p foo put foo1 /etc/passwd 2024-04-25T09:22:08.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:38: TEST_a_merge_empty: rados -p foo put foo2 /etc/passwd 2024-04-25T09:22:08.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:39: TEST_a_merge_empty: rados -p foo put foo3 /etc/passwd 2024-04-25T09:22:08.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:40: TEST_a_merge_empty: rados -p foo put foo4 /etc/passwd 2024-04-25T09:22:09.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:42: TEST_a_merge_empty: rados -p foo put foo5 /etc/passwd 2024-04-25T09:22:09.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:43: TEST_a_merge_empty: rados -p foo put foo6 /etc/passwd 2024-04-25T09:22:09.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:44: TEST_a_merge_empty: rados -p foo put foo8 /etc/passwd 2024-04-25T09:22:09.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:45: TEST_a_merge_empty: rados -p foo put foo10 /etc/passwd 2024-04-25T09:22:09.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:46: TEST_a_merge_empty: rados -p foo put foo11 /etc/passwd 2024-04-25T09:22:10.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:47: TEST_a_merge_empty: rados -p foo put foo12 /etc/passwd 2024-04-25T09:22:10.263 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:48: TEST_a_merge_empty: rados -p foo put foo16 /etc/passwd 2024-04-25T09:22:10.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:50: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:22:10.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:22:10.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:22:10.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:22:10.449 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:22:10.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:22:10.450 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:22:10.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:22:10.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:22:10.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:22:10.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:22:10.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:22:10.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:22:10.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:22:10.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:22:10.570 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:22:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:22:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:22:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:22:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:22:11.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:11.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:22:11.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836513 2024-04-25T09:22:11.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836513 2024-04-25T09:22:11.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513' 2024-04-25T09:22:11.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:11.245 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:22:11.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672980 2024-04-25T09:22:11.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672980 2024-04-25T09:22:11.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-42949672980' 2024-04-25T09:22:11.396 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:11.396 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:22:11.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509454 2024-04-25T09:22:11.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509454 2024-04-25T09:22:11.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-42949672980 2-64424509454' 2024-04-25T09:22:11.550 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:11.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836513 2024-04-25T09:22:11.550 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:11.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:22:11.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836513 2024-04-25T09:22:11.552 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:11.553 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836513 2024-04-25T09:22:11.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836513 2024-04-25T09:22:11.554 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-04-25T09:22:11.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:22:11.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836513 2024-04-25T09:22:11.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:22:12.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:22:12.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:22:13.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836513 -lt 21474836513 2024-04-25T09:22:13.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:13.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672980 2024-04-25T09:22:13.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:13.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:22:13.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672980 2024-04-25T09:22:13.402 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672980 2024-04-25T09:22:13.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-04-25T09:22:13.404 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672980 2024-04-25T09:22:13.404 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:22:13.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672980 -lt 42949672980 2024-04-25T09:22:13.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:13.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:13.872 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509454 2024-04-25T09:22:13.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:22:13.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509454 2024-04-25T09:22:13.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:13.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509454 2024-04-25T09:22:13.875 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509454 2024-04-25T09:22:13.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-04-25T09:22:13.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:22:14.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509454 2024-04-25T09:22:14.336 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:22:14.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:14.337 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:14.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:22:14.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:14.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:14.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:14.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:14.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:14.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:14.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:15.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:22:15.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:15.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:15.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:15.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:22:15.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:22:15.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:22:15.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:52: TEST_a_merge_empty: ceph tell osd.1 config set osd_debug_no_purge_strays true 2024-04-25T09:22:15.985 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T09:22:15.985 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "" 2024-04-25T09:22:15.985 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T09:22:15.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:53: TEST_a_merge_empty: ceph osd pool set foo size 2 2024-04-25T09:22:17.061 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 2 2024-04-25T09:22:17.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:54: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:22:17.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:22:17.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:22:17.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:22:17.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:22:17.087 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:22:17.087 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:22:17.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:22:17.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:22:17.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:22:17.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:22:17.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:22:17.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:22:17.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:22:17.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:22:17.198 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:22:17.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:22:17.776 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:22:17.776 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:22:17.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:22:17.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:17.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:22:17.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836516 2024-04-25T09:22:17.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836516 2024-04-25T09:22:17.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516' 2024-04-25T09:22:17.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:17.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:22:18.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672983 2024-04-25T09:22:18.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672983 2024-04-25T09:22:18.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672983' 2024-04-25T09:22:18.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:18.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:22:18.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509458 2024-04-25T09:22:18.306 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509458 2024-04-25T09:22:18.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672983 2-64424509458' 2024-04-25T09:22:18.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:18.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836516 2024-04-25T09:22:18.307 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:18.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:22:18.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836516 2024-04-25T09:22:18.309 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:18.310 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836516 2024-04-25T09:22:18.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836516 2024-04-25T09:22:18.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-04-25T09:22:18.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:22:18.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836516 -lt 21474836516 2024-04-25T09:22:18.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:18.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672983 2024-04-25T09:22:18.700 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:18.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:22:18.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-04-25T09:22:18.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:18.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672983 2024-04-25T09:22:18.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-04-25T09:22:18.704 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672983 2024-04-25T09:22:18.704 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:22:19.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672983 -lt 42949672983 2024-04-25T09:22:19.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:19.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509458 2024-04-25T09:22:19.101 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:19.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:22:19.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-04-25T09:22:19.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:19.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509458 2024-04-25T09:22:19.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-04-25T09:22:19.105 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509458 2024-04-25T09:22:19.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:22:19.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509458 2024-04-25T09:22:19.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:22:20.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:22:20.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:22:21.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509458 2024-04-25T09:22:21.034 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:22:21.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:21.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:21.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:22:21.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:21.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:21.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:21.554 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:21.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:21.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:21.555 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:22.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:22:22.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:22.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:22.006 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:22.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:22:22.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:22:22.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:22:22.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:56: TEST_a_merge_empty: kill_daemons td/pg-split-merge TERM osd.2 2024-04-25T09:22:22.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:22:22.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:22:22.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:22:22.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:22:22.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:22:22.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:22:22.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:57: TEST_a_merge_empty: ceph-objectstore-tool --data-path td/pg-split-merge/2 --op remove --pgid 1.1 --force 2024-04-25T09:22:24.132 INFO:tasks.workunit.client.0.smithi102.stdout: marking collection for removal 2024-04-25T09:22:24.133 INFO:tasks.workunit.client.0.smithi102.stdout:setting '_remove' omap key 2024-04-25T09:22:24.133 INFO:tasks.workunit.client.0.smithi102.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:22:24.245 INFO:tasks.workunit.client.0.smithi102.stdout:Remove successful 2024-04-25T09:22:24.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:58: TEST_a_merge_empty: activate_osd td/pg-split-merge 2 2024-04-25T09:22:24.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:22:24.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:22:24.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T09:22:24.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:22:24.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/2 2024-04-25T09:22:24.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:22:24.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:22:24.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:22:24.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:22:24.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-04-25T09:22:24.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-04-25T09:22:24.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:22:24.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:22:24.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:22:24.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:22:24.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:22:24.679 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:22:24.680 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:22:24.680 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:22:24.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/2 2024-04-25T09:22:24.682 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:22:24.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T09:22:24.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:22:24.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/2/whoami 2024-04-25T09:22:24.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T09:22:24.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:22:24.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:22:24.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:22:24.723 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:22:24.738+0000 7f175f7a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:24.725 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:22:24.738+0000 7f175f7a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:24.727 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:22:24.742+0000 7f175f7a27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:25.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T09:22:25.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:22:25.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:22:25.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:22:25.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:22:25.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:25.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:22:25.095 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:22:25.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:25.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:25.276 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:22:25.290+0000 7f175f7a27c0 -1 Falling back to public interface 2024-04-25T09:22:25.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:26.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:26.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:26.547 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:22:26.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:22:26.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:26.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:26.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:27.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:27.963 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:27.964 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:22:27.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:22:27.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:27.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:28.151 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:22:28.166+0000 7f175f7a27c0 -1 osd.2 27 log_to_monitors true 2024-04-25T09:22:28.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:22:29.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:22:29.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:22:29.574 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:22:29.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:22:29.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:22:29.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:22:29.988 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 30 up_thru 17 down_at 28 last_clean_interval [15,27) [v2:127.0.0.1:6818/3318480300,v1:127.0.0.1:6819/3318480300] [v2:127.0.0.1:6820/3318480300,v1:127.0.0.1:6821/3318480300] exists,up c306efe5-4b15-4852-baf6-93fe59da434b 2024-04-25T09:22:29.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:22:29.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:22:29.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:22:29.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:60: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:22:29.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:22:29.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:22:29.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:22:29.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:22:29.990 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:22:29.991 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:22:29.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:22:29.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:22:29.991 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:22:30.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:22:30.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:22:30.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:22:30.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:22:30.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:22:30.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:22:30.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:22:30.504 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:22:30.504 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:22:30.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:22:30.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:30.505 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:22:30.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836521 2024-04-25T09:22:30.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836521 2024-04-25T09:22:30.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836521' 2024-04-25T09:22:30.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:30.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:22:30.839 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672988 2024-04-25T09:22:30.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672988 2024-04-25T09:22:30.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836521 1-42949672988' 2024-04-25T09:22:30.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:30.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:22:30.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018883 2024-04-25T09:22:30.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018883 2024-04-25T09:22:30.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836521 1-42949672988 2-128849018883' 2024-04-25T09:22:30.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:30.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836521 2024-04-25T09:22:30.992 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:30.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:22:30.993 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836521 2024-04-25T09:22:30.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:30.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836521 2024-04-25T09:22:30.995 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836521 2024-04-25T09:22:30.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-04-25T09:22:30.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:22:31.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836519 -lt 21474836521 2024-04-25T09:22:31.395 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:22:32.397 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:22:32.397 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:22:32.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836521 -lt 21474836521 2024-04-25T09:22:32.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:32.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672988 2024-04-25T09:22:32.809 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:32.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:22:32.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672988 2024-04-25T09:22:32.811 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:32.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672988 2024-04-25T09:22:32.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-04-25T09:22:32.813 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672988 2024-04-25T09:22:32.813 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:22:33.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672988 -lt 42949672988 2024-04-25T09:22:33.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:33.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-128849018883 2024-04-25T09:22:33.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:33.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:22:33.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-128849018883 2024-04-25T09:22:33.248 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:33.249 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018883 2024-04-25T09:22:33.250 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 128849018883 2024-04-25T09:22:33.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 128849018883' 2024-04-25T09:22:33.250 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:22:33.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018883 -lt 128849018883 2024-04-25T09:22:33.709 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:22:33.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:33.710 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:34.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:22:34.299 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:34.299 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:34.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:34.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:34.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:34.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:34.300 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:34.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:22:34.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:34.786 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:34.787 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:35.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:22:35.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:22:35.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:22:35.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:66: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change true 2024-04-25T09:22:35.476 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T09:22:35.476 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "" 2024-04-25T09:22:35.476 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T09:22:35.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:67: TEST_a_merge_empty: ceph osd out 0 1 2024-04-25T09:22:36.034 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already out. osd.1 is already out. 2024-04-25T09:22:36.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:68: TEST_a_merge_empty: ceph osd pool set foo pg_num 1 2024-04-25T09:22:37.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:69: TEST_a_merge_empty: sleep 5 2024-04-25T09:22:42.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:70: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change false 2024-04-25T09:22:42.190 INFO:tasks.workunit.client.0.smithi102.stdout:{ 2024-04-25T09:22:42.190 INFO:tasks.workunit.client.0.smithi102.stdout: "success": "" 2024-04-25T09:22:42.190 INFO:tasks.workunit.client.0.smithi102.stdout:} 2024-04-25T09:22:42.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:74: TEST_a_merge_empty: ceph osd in 0 1 2024-04-25T09:22:42.732 INFO:tasks.workunit.client.0.smithi102.stderr:osd.0 is already in. osd.1 is already in. 2024-04-25T09:22:42.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:75: TEST_a_merge_empty: ceph osd pool set foo size 3 2024-04-25T09:22:44.184 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 3 2024-04-25T09:22:44.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:77: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:22:44.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:22:44.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:22:44.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:22:44.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:22:44.211 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:22:44.211 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:22:44.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:22:44.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:22:44.211 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:22:44.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:22:44.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:22:44.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:22:44.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:22:44.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:22:44.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:22:44.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:22:44.998 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:22:44.998 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:22:44.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:22:44.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:44.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:22:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836526 2024-04-25T09:22:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836526 2024-04-25T09:22:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526' 2024-04-25T09:22:45.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:45.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:22:45.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672993 2024-04-25T09:22:45.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672993 2024-04-25T09:22:45.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949672993' 2024-04-25T09:22:45.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:45.444 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:22:45.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018888 2024-04-25T09:22:45.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018888 2024-04-25T09:22:45.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-42949672993 2-128849018888' 2024-04-25T09:22:45.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:45.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836526 2024-04-25T09:22:45.625 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:45.626 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:22:45.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836526 2024-04-25T09:22:45.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:45.627 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836526 2024-04-25T09:22:45.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836526 2024-04-25T09:22:45.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-04-25T09:22:45.628 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:22:46.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836523 -lt 21474836526 2024-04-25T09:22:46.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:22:47.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:22:47.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:22:47.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836526 -lt 21474836526 2024-04-25T09:22:47.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:47.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672993 2024-04-25T09:22:47.655 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:22:47.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672993 2024-04-25T09:22:47.656 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:47.656 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672993 2024-04-25T09:22:47.656 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672993 2024-04-25T09:22:47.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-04-25T09:22:47.657 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:22:48.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672993 -lt 42949672993 2024-04-25T09:22:48.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:48.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-128849018888 2024-04-25T09:22:48.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:48.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:22:48.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-128849018888 2024-04-25T09:22:48.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:48.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018888 2024-04-25T09:22:48.088 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 128849018888 2024-04-25T09:22:48.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 128849018888' 2024-04-25T09:22:48.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:22:48.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018888 -lt 128849018888 2024-04-25T09:22:48.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:22:48.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:48.507 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:49.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:22:49.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:49.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:49.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:49.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:49.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:49.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:49.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:49.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:22:49.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:49.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:49.439 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:49.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:22:49.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T09:22:49.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:22:49.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T09:22:49.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:22:49.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:22:50.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:22:50.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:50.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:50.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:50.098 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:50.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:50.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:50.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:50.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:22:50.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:50.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:50.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:51.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:22:51.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:22:51.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:22:51.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:81: TEST_a_merge_empty: ceph pg scrub 1.0 2024-04-25T09:22:51.388 INFO:tasks.workunit.client.0.smithi102.stderr:instructing pg 1.0 on osd.1 to scrub 2024-04-25T09:22:51.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:82: TEST_a_merge_empty: sleep 10 2024-04-25T09:23:01.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:83: TEST_a_merge_empty: ceph log last debug 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:43.265046+0000 mon.a (mon.0) 432 : cluster [DBG] osdmap e38: 3 total, 3 up, 3 in 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:43.748666+0000 mon.a (mon.0) 434 : cluster [DBG] osdmap e39: 3 total, 3 up, 3 in 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:44.286518+0000 mon.a (mon.0) 437 : cluster [DBG] osdmap e40: 3 total, 3 up, 3 in 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:44.183083+0000 mgr.x (mgr.4100) 89 : cluster [DBG] pgmap v111: 2 pgs: 2 unknown; 82 KiB data, 27 MiB used, 100 GiB / 100 GiB avail 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:44.790212+0000 mon.a (mon.0) 438 : cluster [DBG] osdmap e41: 3 total, 3 up, 3 in 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:45.083319+0000 mon.a (mon.0) 441 : cluster [DBG] osdmap e42: 3 total, 3 up, 3 in 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:45.518997+0000 mon.a (mon.0) 442 : cluster [DBG] osdmap e43: 3 total, 3 up, 3 in 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:45.624532+0000 osd.1 (osd.1) 3 : cluster [INF] 1.0 continuing backfill to osd.2 from (24'7,24'7] MIN to 24'7 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:46.557908+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e44: 3 total, 3 up, 3 in 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:46.183709+0000 mgr.x (mgr.4100) 90 : cluster [DBG] pgmap v116: 1 pgs: 1 remapped+peering; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 20 KiB/s wr, 1 op/s 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:47.001774+0000 mon.a (mon.0) 446 : cluster [DBG] osdmap e45: 3 total, 3 up, 3 in 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:48.184101+0000 mgr.x (mgr.4100) 91 : cluster [DBG] pgmap v119: 1 pgs: 1 remapped+peering; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail 2024-04-25T09:23:01.931 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:50.184544+0000 mgr.x (mgr.4100) 94 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 9.2 KiB/s, 0 objects/s recovering 2024-04-25T09:23:01.932 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:51.859860+0000 osd.1 (osd.1) 4 : cluster [DBG] 1.0 scrub starts 2024-04-25T09:23:01.932 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:51.865447+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub ok 2024-04-25T09:23:01.932 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:52.185005+0000 mgr.x (mgr.4100) 99 : cluster [DBG] pgmap v121: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 7.1 KiB/s, 0 objects/s recovering 2024-04-25T09:23:01.932 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:54.185521+0000 mgr.x (mgr.4100) 100 : cluster [DBG] pgmap v122: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 5.9 KiB/s, 0 objects/s recovering 2024-04-25T09:23:01.932 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:56.186068+0000 mgr.x (mgr.4100) 101 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 4.9 KiB/s, 0 objects/s recovering 2024-04-25T09:23:01.932 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:22:58.186558+0000 mgr.x (mgr.4100) 102 : cluster [DBG] pgmap v124: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 4.2 KiB/s, 0 objects/s recovering 2024-04-25T09:23:01.932 INFO:tasks.workunit.client.0.smithi102.stdout:2024-04-25T09:23:00.187074+0000 mgr.x (mgr.4100) 103 : cluster [DBG] pgmap v125: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 3.9 KiB/s, 0 objects/s recovering 2024-04-25T09:23:01.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:84: TEST_a_merge_empty: ceph pg ls 2024-04-25T09:23:02.363 INFO:tasks.workunit.client.0.smithi102.stdout:PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING 2024-04-25T09:23:02.364 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 11 0 0 0 132748 0 0 0 11 active+clean 10s 24'7 45:98 [1,0,2]p1 [1,0,2]p1 2024-04-25T09:22:51.865457+0000 2024-04-25T09:21:53.749936+0000 1 periodic scrub scheduled @ 2024-04-26T12:18:20.462790+0000 2024-04-25T09:23:02.364 INFO:tasks.workunit.client.0.smithi102.stdout: 2024-04-25T09:23:02.364 INFO:tasks.workunit.client.0.smithi102.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:23:02.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: ceph pg ls 2024-04-25T09:23:02.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: grep ' active.clean ' 2024-04-25T09:23:02.767 INFO:tasks.workunit.client.0.smithi102.stdout:1.0 11 0 0 0 132748 0 0 0 11 active+clean 10s 24'7 45:98 [1,0,2]p1 [1,0,2]p1 2024-04-25T09:22:51.865457+0000 2024-04-25T09:21:53.749936+0000 1 periodic scrub scheduled @ 2024-04-26T12:18:20.462790+0000 2024-04-25T09:23:02.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:23:02.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:23:02.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:23:02.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:23:02.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:23:02.768 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:23:02.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:23:02.768 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:23:02.769 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:23:03.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:23:03.080 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:23:03.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:23:03.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:23:03.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:23:03.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:23:03.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:23:03.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:23:03.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:23:03.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:23:03.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:23:03.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:23:03.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:23:03.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:23:03.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:23:03.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:23:03.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:23:03.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.450371 2024-04-25T09:23:03.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:23:03.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:23:03.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:23:03.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:23:03.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:23:03.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:23:03.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:23:03.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:23:03.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:23:03.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:23:03.116 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:23:03.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:23:03.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:23:03.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:23:03.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:23:03.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:23:03.120 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:23:03.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:23:03.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:23:03.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:23:03.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:23:03.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:23:03.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:23:03.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:23:03.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:23:03.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:23:03.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:23:03.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:23:03.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:23:03.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:23:03.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:23:03.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.450371 2024-04-25T09:23:03.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:23:03.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:23:03.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:23:03.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:23:03.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:23:03.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:23:03.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.450371 2024-04-25T09:23:03.133 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:23:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:23:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:23:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:23:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_merge_and_gap td/pg-split-merge 2024-04-25T09:23:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:89: TEST_import_after_merge_and_gap: local dir=td/pg-split-merge 2024-04-25T09:23:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:91: TEST_import_after_merge_and_gap: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:23:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:23:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:23:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:23:03.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:23:03.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:23:03.135 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:23:03.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:23:03.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:23:03.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:23:03.642 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:23:03.642 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:23:03.642 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:23:03.642 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:23:03.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:23:03.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:23:03.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:23:03.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:23:03.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:23:03.853 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:23:03.855 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:23:03.855 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:23:03.855 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:23:03.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:23:03.856 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:23:03.856 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:23:03.856 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:23:03.856 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.450371/ceph-mon.a.asok 2024-04-25T09:23:03.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:23:03.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.450371/ceph-mon.a.asok config get fsid 2024-04-25T09:23:03.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:23:03.967 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:23:03.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:23:03.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:23:03.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:23:03.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:23:03.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:23:03.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:23:03.969 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:23:03.969 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:23:03.969 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:23:03.969 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.450371/ceph-mon.a.asok 2024-04-25T09:23:03.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:23:03.969 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.450371/ceph-mon.a.asok config get mon_host 2024-04-25T09:23:04.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:92: TEST_import_after_merge_and_gap: run_mgr td/pg-split-merge x 2024-04-25T09:23:04.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:23:04.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:23:04.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:23:04.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:23:04.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:23:04.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:23:04.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:23:04.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:23:04.989 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:23:04.989 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:23:04.990 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:23:04.990 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:23:04.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:23:04.990 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:23:04.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:23:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:93: TEST_import_after_merge_and_gap: run_osd td/pg-split-merge 0 2024-04-25T09:23:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:23:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:23:05.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:23:05.028 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:23:05.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:23:05.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:23:05.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:23:05.030 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:23:05.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=150f6751-a5c0-42ad-b859-7e7a4382302c 2024-04-25T09:23:05.030 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 150f6751-a5c0-42ad-b859-7e7a4382302c 2024-04-25T09:23:05.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 150f6751-a5c0-42ad-b859-7e7a4382302c' 2024-04-25T09:23:05.031 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:23:05.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5ICpmVmbHAxAAvxd4MVo2Dn+iWvD5sYh4GQ== 2024-04-25T09:23:05.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5ICpmVmbHAxAAvxd4MVo2Dn+iWvD5sYh4GQ=="}' 2024-04-25T09:23:05.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 150f6751-a5c0-42ad-b859-7e7a4382302c -i td/pg-split-merge/0/new.json 2024-04-25T09:23:05.354 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:23:05.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:23:05.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5ICpmVmbHAxAAvxd4MVo2Dn+iWvD5sYh4GQ== --osd-uuid 150f6751-a5c0-42ad-b859-7e7a4382302c 2024-04-25T09:23:05.406 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:05.421+0000 7f738b2947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:05.423 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:05.437+0000 7f738b2947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:05.424 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:05.437+0000 7f738b2947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:05.425 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:05.437+0000 7f738b2947c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:23:05.425 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:05.437+0000 7f738b2947c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:23:05.425 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:05.437+0000 7f738b2947c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:23:05.463 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:05.477+0000 7f738b2947c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:23:11.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:23:11.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:23:11.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:23:11.810 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:23:11.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:23:12.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:23:12.376 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:23:12.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:23:12.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:23:12.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:23:12.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:23:12.405 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:12.417+0000 7fb5babf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:12.407 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:12.421+0000 7fb5babf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:12.409 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:12.421+0000 7fb5babf77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:12.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:23:12.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:23:12.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:23:12.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:23:12.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:23:12.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:12.833 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:23:12.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:23:12.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:12.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:13.000 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:13.013+0000 7fb5babf77c0 -1 Falling back to public interface 2024-04-25T09:23:13.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:14.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:14.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:14.269 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:23:14.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:23:14.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:14.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:14.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:15.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:15.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:15.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:23:15.675 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:23:15.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:15.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:15.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:15.701+0000 7fb5babf77c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:23:16.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:17.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:17.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:17.286 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:23:17.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:23:17.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:17.287 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:17.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:18.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:18.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:18.752 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:23:18.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:23:18.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:18.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:19.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:20.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:20.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:20.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:23:20.183 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:23:20.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:20.183 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:20.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:21.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:21.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:21.719 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:23:21.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:23:21.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:21.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:22.185 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:23.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:23.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:23.187 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:23:23.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:23:23.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:23.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:23.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:24.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:24.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:24.780 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:23:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:23:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:24.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:25.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:26.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:26.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:26.213 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:23:26.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:23:26.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:26.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:26.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:27.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:27.698 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:27.698 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:23:27.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:23:27.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:27.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:28.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:29.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:29.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:29.142 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:23:29.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:23:29.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:29.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:29.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:30.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:30.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:30.669 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:23:30.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:23:30.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:30.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:31.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:32.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:32.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:32.753 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:23:32.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:23:32.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:32.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:33.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:34.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:34.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:34.513 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:23:34.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:23:34.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:34.514 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:35.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:36.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:36.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:36.376 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:23:36.377 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:23:36.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:36.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:37.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:38.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:38.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:38.357 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:23:38.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:23:38.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:38.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:39.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:40.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:40.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:40.307 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:23:40.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:23:40.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:40.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:40.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:41.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:41.914 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:41.914 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:23:41.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:23:41.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:41.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:42.329 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3654682586,v1:127.0.0.1:6803/3654682586] [v2:127.0.0.1:6804/3654682586,v1:127.0.0.1:6805/3654682586] exists,up 150f6751-a5c0-42ad-b859-7e7a4382302c 2024-04-25T09:23:42.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:23:42.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:23:42.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:23:42.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:95: TEST_import_after_merge_and_gap: ceph osd pool create foo 2 2024-04-25T09:23:42.888 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T09:23:42.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:96: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:23:42.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:23:42.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:23:42.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:23:42.906 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:23:42.907 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:23:42.907 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:23:42.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:23:42.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:23:42.907 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:23:43.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:23:43.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:23:43.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:23:43.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:23:43.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:23:43.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:23:43.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:23:43.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:23:43.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:23:43.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:23:43.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836494 2024-04-25T09:23:43.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836494 2024-04-25T09:23:43.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836494' 2024-04-25T09:23:43.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:23:43.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836494 2024-04-25T09:23:43.753 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:23:43.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:23:43.754 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836494 2024-04-25T09:23:43.755 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:23:43.756 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836494 2024-04-25T09:23:43.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836494 2024-04-25T09:23:43.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-04-25T09:23:43.756 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:23:44.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836494 2024-04-25T09:23:44.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:23:45.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:23:45.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:23:45.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836494 -lt 21474836494 2024-04-25T09:23:45.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:23:45.780 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:23:45.781 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:23:46.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:23:46.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:23:46.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:23:46.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:23:46.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:23:46.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:23:46.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:23:46.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:23:46.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:23:46.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:23:46.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:23:46.782 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:23:47.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:23:47.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:23:47.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:23:47.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:97: TEST_import_after_merge_and_gap: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-04-25T09:23:47.396 INFO:tasks.workunit.client.0.smithi102.stdout:hints = 1 2024-04-25T09:23:47.396 INFO:tasks.workunit.client.0.smithi102.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-04-25T09:23:47.396 INFO:tasks.workunit.client.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_461693 2024-04-25T09:23:47.397 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:23:47.397 INFO:tasks.workunit.client.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:23:48.396 INFO:tasks.workunit.client.0.smithi102.stdout: 1 16 97 81 0.079093 0.0791016 0.193041 0.178824 2024-04-25T09:23:49.397 INFO:tasks.workunit.client.0.smithi102.stdout: 2 16 209 193 0.0942262 0.109375 0.142249 0.15547 2024-04-25T09:23:50.397 INFO:tasks.workunit.client.0.smithi102.stdout: 3 16 320 304 0.0989442 0.108398 0.186572 0.154713 2024-04-25T09:23:51.397 INFO:tasks.workunit.client.0.smithi102.stdout:Total time run: 3.07914 2024-04-25T09:23:51.397 INFO:tasks.workunit.client.0.smithi102.stdout:Total writes made: 320 2024-04-25T09:23:51.397 INFO:tasks.workunit.client.0.smithi102.stdout:Write size: 1024 2024-04-25T09:23:51.397 INFO:tasks.workunit.client.0.smithi102.stdout:Object size: 1024 2024-04-25T09:23:51.397 INFO:tasks.workunit.client.0.smithi102.stdout:Bandwidth (MB/sec): 0.101489 2024-04-25T09:23:51.398 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Bandwidth: 0.0172034 2024-04-25T09:23:51.398 INFO:tasks.workunit.client.0.smithi102.stdout:Max bandwidth (MB/sec): 0.109375 2024-04-25T09:23:51.398 INFO:tasks.workunit.client.0.smithi102.stdout:Min bandwidth (MB/sec): 0.0791016 2024-04-25T09:23:51.398 INFO:tasks.workunit.client.0.smithi102.stdout:Average IOPS: 103 2024-04-25T09:23:51.398 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev IOPS: 17.6163 2024-04-25T09:23:51.398 INFO:tasks.workunit.client.0.smithi102.stdout:Max IOPS: 112 2024-04-25T09:23:51.398 INFO:tasks.workunit.client.0.smithi102.stdout:Min IOPS: 81 2024-04-25T09:23:51.398 INFO:tasks.workunit.client.0.smithi102.stdout:Average Latency(s): 0.1537 2024-04-25T09:23:51.398 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Latency(s): 0.0344473 2024-04-25T09:23:51.398 INFO:tasks.workunit.client.0.smithi102.stdout:Max latency(s): 0.23933 2024-04-25T09:23:51.398 INFO:tasks.workunit.client.0.smithi102.stdout:Min latency(s): 0.116096 2024-04-25T09:23:51.471 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:99: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:23:51.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:23:51.472 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:23:51.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:23:51.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:23:51.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:23:51.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:23:51.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:100: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-04-25T09:23:52.042 INFO:tasks.workunit.client.0.smithi102.stderr:Exporting 1.1 info 1.1( v 10'145 (0'0,10'145] local-lis/les=7/8 n=145 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:23:52.046 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:80decefa:::benchmark_data_smithi102_461693_object215:head# 2024-04-25T09:23:52.049 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:82504851:::benchmark_data_smithi102_461693_object59:head# 2024-04-25T09:23:52.049 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:82bc124c:::benchmark_data_smithi102_461693_object242:head# 2024-04-25T09:23:52.050 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:883cb801:::benchmark_data_smithi102_461693_object150:head# 2024-04-25T09:23:52.056 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:889e9aa5:::benchmark_data_smithi102_461693_object250:head# 2024-04-25T09:23:52.058 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:890d9aaf:::benchmark_data_smithi102_461693_object96:head# 2024-04-25T09:23:52.059 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8a093910:::benchmark_data_smithi102_461693_object15:head# 2024-04-25T09:23:52.060 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8aedfdc0:::benchmark_data_smithi102_461693_object193:head# 2024-04-25T09:23:52.060 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8afeae38:::benchmark_data_smithi102_461693_object221:head# 2024-04-25T09:23:52.061 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8b078005:::benchmark_data_smithi102_461693_object60:head# 2024-04-25T09:23:52.061 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8b4ee6c4:::benchmark_data_smithi102_461693_object166:head# 2024-04-25T09:23:52.061 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8ca6a195:::benchmark_data_smithi102_461693_object172:head# 2024-04-25T09:23:52.061 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8cdff5e0:::benchmark_data_smithi102_461693_object17:head# 2024-04-25T09:23:52.063 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8d571cff:::benchmark_data_smithi102_461693_object255:head# 2024-04-25T09:23:52.063 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8eeaf510:::benchmark_data_smithi102_461693_object22:head# 2024-04-25T09:23:52.063 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8f3dbf2e:::benchmark_data_smithi102_461693_object110:head# 2024-04-25T09:23:52.063 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:91d0aef9:::benchmark_data_smithi102_461693_object132:head# 2024-04-25T09:23:52.063 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:93bbd15e:::benchmark_data_smithi102_461693_object311:head# 2024-04-25T09:23:52.063 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:942bc514:::benchmark_data_smithi102_461693_object77:head# 2024-04-25T09:23:52.064 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:94d0ab29:::benchmark_data_smithi102_461693_object285:head# 2024-04-25T09:23:52.065 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:94ec3b32:::benchmark_data_smithi102_461693_object72:head# 2024-04-25T09:23:52.065 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:95092f81:::benchmark_data_smithi102_461693_object220:head# 2024-04-25T09:23:52.065 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:95b06a69:::benchmark_data_smithi102_461693_object67:head# 2024-04-25T09:23:52.066 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:96ade7c2:::benchmark_data_smithi102_461693_object115:head# 2024-04-25T09:23:52.066 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:971cae83:::benchmark_data_smithi102_461693_object196:head# 2024-04-25T09:23:52.066 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:978b03f9:::benchmark_data_smithi102_461693_object236:head# 2024-04-25T09:23:52.066 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:98701baa:::benchmark_data_smithi102_461693_object55:head# 2024-04-25T09:23:52.067 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:98e4a738:::benchmark_data_smithi102_461693_object127:head# 2024-04-25T09:23:52.068 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9aa9891c:::benchmark_data_smithi102_461693_object5:head# 2024-04-25T09:23:52.068 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9afe2884:::benchmark_data_smithi102_461693_object273:head# 2024-04-25T09:23:52.069 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9bd5d2e8:::benchmark_data_smithi102_461693_object265:head# 2024-04-25T09:23:52.070 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9bf31ddf:::benchmark_data_smithi102_461693_object83:head# 2024-04-25T09:23:52.071 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9c9b760d:::benchmark_data_smithi102_461693_object291:head# 2024-04-25T09:23:52.072 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9c9c7e9b:::benchmark_data_smithi102_461693_object70:head# 2024-04-25T09:23:52.072 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9cc82558:::benchmark_data_smithi102_461693_object108:head# 2024-04-25T09:23:52.073 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9e60200b:::benchmark_data_smithi102_461693_object26:head# 2024-04-25T09:23:52.074 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9ebdc7a5:::benchmark_data_smithi102_461693_object18:head# 2024-04-25T09:23:52.074 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9eccdf33:::benchmark_data_smithi102_461693_object131:head# 2024-04-25T09:23:52.075 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9fcd1944:::benchmark_data_smithi102_461693_object198:head# 2024-04-25T09:23:52.075 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9fcee8c9:::benchmark_data_smithi102_461693_object263:head# 2024-04-25T09:23:52.076 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a109394d:::benchmark_data_smithi102_461693_object49:head# 2024-04-25T09:23:52.076 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a284d2b1:::benchmark_data_smithi102_461693_object40:head# 2024-04-25T09:23:52.077 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a2ff1fbf:::benchmark_data_smithi102_461693_object52:head# 2024-04-25T09:23:52.077 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a32db852:::benchmark_data_smithi102_461693_object232:head# 2024-04-25T09:23:52.078 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a4f4d3a4:::benchmark_data_smithi102_461693_object261:head# 2024-04-25T09:23:52.078 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a7538180:::benchmark_data_smithi102_461693_object308:head# 2024-04-25T09:23:52.080 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:aa0c2ca6:::benchmark_data_smithi102_461693_object213:head# 2024-04-25T09:23:52.080 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:aa2b501d:::benchmark_data_smithi102_461693_object218:head# 2024-04-25T09:23:52.080 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:aae249da:::benchmark_data_smithi102_461693_object182:head# 2024-04-25T09:23:52.081 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac080002:::benchmark_data_smithi102_461693_object13:head# 2024-04-25T09:23:52.082 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac64182c:::benchmark_data_smithi102_461693_object295:head# 2024-04-25T09:23:52.083 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac96a035:::benchmark_data_smithi102_461693_object46:head# 2024-04-25T09:23:52.083 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:acd4c3d0:::benchmark_data_smithi102_461693_object226:head# 2024-04-25T09:23:52.084 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ad92f4ab:::benchmark_data_smithi102_461693_object303:head# 2024-04-25T09:23:52.084 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:adef340d:::benchmark_data_smithi102_461693_object61:head# 2024-04-25T09:23:52.110 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b1d35c85:::benchmark_data_smithi102_461693_object200:head# 2024-04-25T09:23:52.118 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b1d73141:::benchmark_data_smithi102_461693_object112:head# 2024-04-25T09:23:52.119 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b4852218:::benchmark_data_smithi102_461693_object277:head# 2024-04-25T09:23:52.144 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b56c2919:::benchmark_data_smithi102_461693_object270:head# 2024-04-25T09:23:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b56e0ab2:::benchmark_data_smithi102_461693_object231:head# 2024-04-25T09:23:52.152 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b5c35fa0:::benchmark_data_smithi102_461693_object130:head# 2024-04-25T09:23:52.169 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b7e1813e:::benchmark_data_smithi102_461693_object185:head# 2024-04-25T09:23:52.177 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:23:52.185 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b8772720:::benchmark_data_smithi102_461693_object289:head# 2024-04-25T09:23:52.194 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bba685cf:::benchmark_data_smithi102_461693_object155:head# 2024-04-25T09:23:52.194 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bc36f44b:::benchmark_data_smithi102_461693_object296:head# 2024-04-25T09:23:52.195 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bc9448f0:::benchmark_data_smithi102_461693_object82:head# 2024-04-25T09:23:52.196 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bd20be31:::benchmark_data_smithi102_461693_object93:head# 2024-04-25T09:23:52.196 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bda0949f:::benchmark_data_smithi102_461693_object181:head# 2024-04-25T09:23:52.197 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bde44e2f:::benchmark_data_smithi102_461693_object95:head# 2024-04-25T09:23:52.197 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:be6d30be:::benchmark_data_smithi102_461693_object207:head# 2024-04-25T09:23:52.198 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bf7d046b:::benchmark_data_smithi102_461693_object278:head# 2024-04-25T09:23:52.199 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c1eaa85c:::benchmark_data_smithi102_461693_object137:head# 2024-04-25T09:23:52.199 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c37f1115:::benchmark_data_smithi102_461693_object310:head# 2024-04-25T09:23:52.200 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c50db599:::benchmark_data_smithi102_461693_object176:head# 2024-04-25T09:23:52.201 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c53ef393:::benchmark_data_smithi102_461693_object279:head# 2024-04-25T09:23:52.202 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c53f44d8:::benchmark_data_smithi102_461693_object239:head# 2024-04-25T09:23:52.203 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c551c4c0:::benchmark_data_smithi102_461693_object123:head# 2024-04-25T09:23:52.203 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c55d9b9d:::benchmark_data_smithi102_461693_object237:head# 2024-04-25T09:23:52.204 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c6ad85a0:::benchmark_data_smithi102_461693_object41:head# 2024-04-25T09:23:52.204 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c6d379b1:::benchmark_data_smithi102_461693_object304:head# 2024-04-25T09:23:52.205 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c74b1c18:::benchmark_data_smithi102_461693_object113:head# 2024-04-25T09:23:52.206 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c7f88b4e:::benchmark_data_smithi102_461693_object284:head# 2024-04-25T09:23:52.206 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c88cd082:::benchmark_data_smithi102_461693_object259:head# 2024-04-25T09:23:52.207 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c8e2e872:::benchmark_data_smithi102_461693_object39:head# 2024-04-25T09:23:52.208 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c93a1b98:::benchmark_data_smithi102_461693_object142:head# 2024-04-25T09:23:52.208 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c9c320e5:::benchmark_data_smithi102_461693_object249:head# 2024-04-25T09:23:52.209 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ca288c06:::benchmark_data_smithi102_461693_object153:head# 2024-04-25T09:23:52.210 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cba55ca9:::benchmark_data_smithi102_461693_object283:head# 2024-04-25T09:23:52.210 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cbd30596:::benchmark_data_smithi102_461693_object145:head# 2024-04-25T09:23:52.211 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cc3765a4:::benchmark_data_smithi102_461693_object0:head# 2024-04-25T09:23:52.212 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ce561cbb:::benchmark_data_smithi102_461693_object244:head# 2024-04-25T09:23:52.212 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cec1e000:::benchmark_data_smithi102_461693_object171:head# 2024-04-25T09:23:52.213 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d07784f4:::benchmark_data_smithi102_461693_object107:head# 2024-04-25T09:23:52.214 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d131843b:::benchmark_data_smithi102_461693_object119:head# 2024-04-25T09:23:52.214 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d1f2e35b:::benchmark_data_smithi102_461693_object257:head# 2024-04-25T09:23:52.215 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d2bb2fe0:::benchmark_data_smithi102_461693_object197:head# 2024-04-25T09:23:52.216 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d2f5066b:::benchmark_data_smithi102_461693_object307:head# 2024-04-25T09:23:52.216 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d77f7f03:::benchmark_data_smithi102_461693_object170:head# 2024-04-25T09:23:52.217 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d95e4094:::benchmark_data_smithi102_461693_object105:head# 2024-04-25T09:23:52.218 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:da23c414:::benchmark_data_smithi102_461693_object57:head# 2024-04-25T09:23:52.218 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:da5b1f76:::benchmark_data_smithi102_461693_object73:head# 2024-04-25T09:23:52.219 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:da5b6865:::benchmark_data_smithi102_461693_object290:head# 2024-04-25T09:23:52.220 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:db2f70fa:::benchmark_data_smithi102_461693_object230:head# 2024-04-25T09:23:52.220 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dbf5b9d1:::benchmark_data_smithi102_461693_object312:head# 2024-04-25T09:23:52.221 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:de452fc5:::benchmark_data_smithi102_461693_object167:head# 2024-04-25T09:23:52.222 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e195d889:::benchmark_data_smithi102_461693_object129:head# 2024-04-25T09:23:52.222 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e2323390:::benchmark_data_smithi102_461693_object33:head# 2024-04-25T09:23:52.223 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e2b7382e:::benchmark_data_smithi102_461693_object210:head# 2024-04-25T09:23:52.224 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e2bbe377:::benchmark_data_smithi102_461693_object114:head# 2024-04-25T09:23:52.224 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e301fcee:::benchmark_data_smithi102_461693_object62:head# 2024-04-25T09:23:52.225 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e302dfe1:::benchmark_data_smithi102_461693_object227:head# 2024-04-25T09:23:52.226 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e4324c30:::benchmark_data_smithi102_461693_object158:head# 2024-04-25T09:23:52.226 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e447aef9:::benchmark_data_smithi102_461693_object180:head# 2024-04-25T09:23:52.227 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e4558303:::benchmark_data_smithi102_461693_object4:head# 2024-04-25T09:23:52.227 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e5053873:::benchmark_data_smithi102_461693_object81:head# 2024-04-25T09:23:52.228 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e5210074:::benchmark_data_smithi102_461693_object3:head# 2024-04-25T09:23:52.228 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e7382a9c:::benchmark_data_smithi102_461693_object299:head# 2024-04-25T09:23:52.229 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e819bcc3:::benchmark_data_smithi102_461693_object84:head# 2024-04-25T09:23:52.229 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e8d6bf29:::benchmark_data_smithi102_461693_object101:head# 2024-04-25T09:23:52.230 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ea24f1cb:::benchmark_data_smithi102_461693_object186:head# 2024-04-25T09:23:52.231 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ec9bc09f:::benchmark_data_smithi102_461693_object229:head# 2024-04-25T09:23:52.231 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ecca2721:::benchmark_data_smithi102_461693_object134:head# 2024-04-25T09:23:52.231 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ecdd65cb:::benchmark_data_smithi102_461693_object269:head# 2024-04-25T09:23:52.232 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:eed87a51:::benchmark_data_smithi102_461693_object191:head# 2024-04-25T09:23:52.233 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ef8637ba:::benchmark_data_smithi102_461693_object151:head# 2024-04-25T09:23:52.234 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f02b8d22:::benchmark_data_smithi102_461693_object168:head# 2024-04-25T09:23:52.234 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f06a39d9:::benchmark_data_smithi102_461693_object275:head# 2024-04-25T09:23:52.235 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f136e2f9:::benchmark_data_smithi102_461693_object267:head# 2024-04-25T09:23:52.235 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f2467763:::benchmark_data_smithi102_461693_object156:head# 2024-04-25T09:23:52.236 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f3685b12:::benchmark_data_smithi102_461693_object212:head# 2024-04-25T09:23:52.237 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f39374a5:::benchmark_data_smithi102_461693_object184:head# 2024-04-25T09:23:52.237 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f5c19b84:::benchmark_data_smithi102_461693_object225:head# 2024-04-25T09:23:52.238 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f69416ea:::benchmark_data_smithi102_461693_object235:head# 2024-04-25T09:23:52.238 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f7295629:::benchmark_data_smithi102_461693_object87:head# 2024-04-25T09:23:52.239 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f90d383f:::benchmark_data_smithi102_461693_object203:head# 2024-04-25T09:23:52.240 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f939e8fc:::benchmark_data_smithi102_461693_object48:head# 2024-04-25T09:23:52.240 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f993a43e:::benchmark_data_smithi102_461693_object99:head# 2024-04-25T09:23:52.241 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f9fdbd9c:::benchmark_data_smithi102_461693_object86:head# 2024-04-25T09:23:52.241 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fa2a1c9b:::benchmark_data_smithi102_461693_object47:head# 2024-04-25T09:23:52.242 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fb939772:::benchmark_data_smithi102_461693_object9:head# 2024-04-25T09:23:52.242 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fca3284e:::benchmark_data_smithi102_461693_object187:head# 2024-04-25T09:23:52.243 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fe98a8ca:::benchmark_data_smithi102_461693_object258:head# 2024-04-25T09:23:52.243 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ff7040e0:::benchmark_data_smithi102_461693_object281:head# 2024-04-25T09:23:52.244 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ffab2f9e:::benchmark_data_smithi102_461693_object154:head# 2024-04-25T09:23:52.245 INFO:tasks.workunit.client.0.smithi102.stderr:Export successful 2024-04-25T09:23:52.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:101: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:23:52.652 INFO:tasks.workunit.client.0.smithi102.stderr:Exporting 1.0 info 1.0( v 10'176 (0'0,10'176] local-lis/les=7/8 n=176 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:23:52.663 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:000b3b92:::benchmark_data_smithi102_461693_object262:head# 2024-04-25T09:23:52.663 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:00374c2e:::benchmark_data_smithi102_461693_object216:head# 2024-04-25T09:23:52.664 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0301c80c:::benchmark_data_smithi102_461693_object11:head# 2024-04-25T09:23:52.664 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:036651e6:::benchmark_data_smithi102_461693_object111:head# 2024-04-25T09:23:52.665 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0402916e:::benchmark_data_smithi102_461693_object183:head# 2024-04-25T09:23:52.666 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:046b3940:::benchmark_data_smithi102_461693_object104:head# 2024-04-25T09:23:52.666 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0540312f:::benchmark_data_smithi102_461693_object69:head# 2024-04-25T09:23:52.666 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:054ee2e1:::benchmark_data_smithi102_461693_object300:head# 2024-04-25T09:23:52.668 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:069b73e0:::benchmark_data_smithi102_461693_object195:head# 2024-04-25T09:23:52.668 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:07526ebf:::benchmark_data_smithi102_461693_object248:head# 2024-04-25T09:23:52.668 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:08aeff1a:::benchmark_data_smithi102_461693_object43:head# 2024-04-25T09:23:52.668 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:09569df1:::benchmark_data_smithi102_461693_object305:head# 2024-04-25T09:23:52.668 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:099020c4:::benchmark_data_smithi102_461693_object152:head# 2024-04-25T09:23:52.669 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0a47cb3d:::benchmark_data_smithi102_461693_object266:head# 2024-04-25T09:23:52.669 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0a7de0f7:::benchmark_data_smithi102_461693_object208:head# 2024-04-25T09:23:52.669 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0b6a70be:::benchmark_data_smithi102_461693_object85:head# 2024-04-25T09:23:52.670 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0cc78a03:::benchmark_data_smithi102_461693_object21:head# 2024-04-25T09:23:52.678 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0cd9cd5d:::benchmark_data_smithi102_461693_object160:head# 2024-04-25T09:23:52.679 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0deb35d3:::benchmark_data_smithi102_461693_object217:head# 2024-04-25T09:23:52.679 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0e8b2dd0:::benchmark_data_smithi102_461693_object148:head# 2024-04-25T09:23:52.681 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0f24b073:::benchmark_data_smithi102_461693_object317:head# 2024-04-25T09:23:52.681 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:10af48fb:::benchmark_data_smithi102_461693_object240:head# 2024-04-25T09:23:52.683 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:11692bdb:::benchmark_data_smithi102_461693_object27:head# 2024-04-25T09:23:52.683 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:128e9745:::benchmark_data_smithi102_461693_object314:head# 2024-04-25T09:23:52.684 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:132fbbae:::benchmark_data_smithi102_461693_object92:head# 2024-04-25T09:23:52.685 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:14a6a0cd:::benchmark_data_smithi102_461693_object88:head# 2024-04-25T09:23:52.686 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1512b06b:::benchmark_data_smithi102_461693_object272:head# 2024-04-25T09:23:52.686 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1629e659:::benchmark_data_smithi102_461693_object256:head# 2024-04-25T09:23:52.687 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:16f7bd8c:::benchmark_data_smithi102_461693_object64:head# 2024-04-25T09:23:52.687 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1786312c:::benchmark_data_smithi102_461693_object37:head# 2024-04-25T09:23:52.688 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:19fa802a:::benchmark_data_smithi102_461693_object177:head# 2024-04-25T09:23:52.688 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1d2fbc72:::benchmark_data_smithi102_461693_object253:head# 2024-04-25T09:23:52.689 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1dbe156c:::benchmark_data_smithi102_461693_object301:head# 2024-04-25T09:23:52.689 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1e43a575:::benchmark_data_smithi102_461693_object109:head# 2024-04-25T09:23:52.690 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1e98b968:::benchmark_data_smithi102_461693_object8:head# 2024-04-25T09:23:52.691 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1e9976c2:::benchmark_data_smithi102_461693_object31:head# 2024-04-25T09:23:52.691 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:20ac13d7:::benchmark_data_smithi102_461693_object319:head# 2024-04-25T09:23:52.692 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2279d77a:::benchmark_data_smithi102_461693_object157:head# 2024-04-25T09:23:52.692 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:22b40e23:::benchmark_data_smithi102_461693_object233:head# 2024-04-25T09:23:52.693 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:23111692:::benchmark_data_smithi102_461693_object14:head# 2024-04-25T09:23:52.694 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:23426916:::benchmark_data_smithi102_461693_object164:head# 2024-04-25T09:23:52.694 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:238a3ae3:::benchmark_data_smithi102_461693_object211:head# 2024-04-25T09:23:52.695 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:239e27b3:::benchmark_data_smithi102_461693_object219:head# 2024-04-25T09:23:52.695 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:23b4b493:::benchmark_data_smithi102_461693_object192:head# 2024-04-25T09:23:52.696 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:24e58257:::benchmark_data_smithi102_461693_object141:head# 2024-04-25T09:23:52.697 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:24e6a109:::benchmark_data_smithi102_461693_object106:head# 2024-04-25T09:23:52.697 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:25997359:::benchmark_data_smithi102_461693_object264:head# 2024-04-25T09:23:52.698 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:25b16926:::benchmark_data_smithi102_461693_object29:head# 2024-04-25T09:23:52.699 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:26824afa:::benchmark_data_smithi102_461693_object94:head# 2024-04-25T09:23:52.700 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:26a55470:::benchmark_data_smithi102_461693_object274:head# 2024-04-25T09:23:52.700 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:272eb2c9:::benchmark_data_smithi102_461693_object149:head# 2024-04-25T09:23:52.701 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:273627fb:::benchmark_data_smithi102_461693_object125:head# 2024-04-25T09:23:52.701 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:273b3e12:::benchmark_data_smithi102_461693_object161:head# 2024-04-25T09:23:52.702 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:275d130a:::benchmark_data_smithi102_461693_object20:head# 2024-04-25T09:23:52.702 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:28694220:::benchmark_data_smithi102_461693_object136:head# 2024-04-25T09:23:52.703 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2a0bbe58:::benchmark_data_smithi102_461693_object199:head# 2024-04-25T09:23:52.704 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2afc2f48:::benchmark_data_smithi102_461693_object209:head# 2024-04-25T09:23:52.704 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2bbb9370:::benchmark_data_smithi102_461693_object287:head# 2024-04-25T09:23:52.705 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2bd520cd:::benchmark_data_smithi102_461693_object19:head# 2024-04-25T09:23:52.705 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2be202af:::benchmark_data_smithi102_461693_object234:head# 2024-04-25T09:23:52.706 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2c0b4651:::benchmark_data_smithi102_461693_object74:head# 2024-04-25T09:23:52.707 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2c9a727c:::benchmark_data_smithi102_461693_object165:head# 2024-04-25T09:23:52.707 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2d3d0890:::benchmark_data_smithi102_461693_object50:head# 2024-04-25T09:23:52.709 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2d3fef0a:::benchmark_data_smithi102_461693_object173:head# 2024-04-25T09:23:52.709 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2d4d8992:::benchmark_data_smithi102_461693_object53:head# 2024-04-25T09:23:52.710 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2ee095c4:::benchmark_data_smithi102_461693_object58:head# 2024-04-25T09:23:52.710 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2fd07b30:::benchmark_data_smithi102_461693_object194:head# 2024-04-25T09:23:52.710 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2fee0198:::benchmark_data_smithi102_461693_object2:head# 2024-04-25T09:23:52.711 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:34e9f16c:::benchmark_data_smithi102_461693_object260:head# 2024-04-25T09:23:52.711 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:355453a0:::benchmark_data_smithi102_461693_object118:head# 2024-04-25T09:23:52.712 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:363cd543:::benchmark_data_smithi102_461693_object34:head# 2024-04-25T09:23:52.712 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:36551f3c:::benchmark_data_smithi102_461693_object51:head# 2024-04-25T09:23:52.712 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3693711a:::benchmark_data_smithi102_461693_object306:head# 2024-04-25T09:23:52.713 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:37225bcc:::benchmark_data_smithi102_461693_object117:head# 2024-04-25T09:23:52.713 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:37c0cca3:::benchmark_data_smithi102_461693_object282:head# 2024-04-25T09:23:52.714 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:389b2f68:::benchmark_data_smithi102_461693_object54:head# 2024-04-25T09:23:52.714 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:38f5ec38:::benchmark_data_smithi102_461693_object12:head# 2024-04-25T09:23:52.715 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:39a2e424:::benchmark_data_smithi102_461693_object16:head# 2024-04-25T09:23:52.715 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3b72c6d5:::benchmark_data_smithi102_461693_object28:head# 2024-04-25T09:23:52.715 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3bca4278:::benchmark_data_smithi102_461693_object6:head# 2024-04-25T09:23:52.716 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3c78f5d8:::benchmark_data_smithi102_461693_object124:head# 2024-04-25T09:23:52.716 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3ccb0f2b:::benchmark_data_smithi102_461693_object144:head# 2024-04-25T09:23:52.717 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3d133806:::benchmark_data_smithi102_461693_object243:head# 2024-04-25T09:23:52.717 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3d2ad51e:::benchmark_data_smithi102_461693_object38:head# 2024-04-25T09:23:52.718 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3d611f0a:::benchmark_data_smithi102_461693_object32:head# 2024-04-25T09:23:52.719 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3f6d6ad9:::benchmark_data_smithi102_461693_object204:head# 2024-04-25T09:23:52.719 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3f7d2427:::benchmark_data_smithi102_461693_object205:head# 2024-04-25T09:23:52.720 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:40581110:::benchmark_data_smithi102_461693_object147:head# 2024-04-25T09:23:52.720 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4066b8f9:::benchmark_data_smithi102_461693_object103:head# 2024-04-25T09:23:52.721 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:407cca9e:::benchmark_data_smithi102_461693_object228:head# 2024-04-25T09:23:52.722 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:411befe3:::benchmark_data_smithi102_461693_object25:head# 2024-04-25T09:23:52.722 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:412d23d7:::benchmark_data_smithi102_461693_object174:head# 2024-04-25T09:23:52.723 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:426f5982:::benchmark_data_smithi102_461693_object7:head# 2024-04-25T09:23:52.723 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:443e3636:::benchmark_data_smithi102_461693_object201:head# 2024-04-25T09:23:52.724 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:45dbb99e:::benchmark_data_smithi102_461693_object139:head# 2024-04-25T09:23:52.725 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:462c230e:::benchmark_data_smithi102_461693_object189:head# 2024-04-25T09:23:52.725 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:46e420c2:::benchmark_data_smithi102_461693_object315:head# 2024-04-25T09:23:52.726 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:46ed67bb:::benchmark_data_smithi102_461693_object80:head# 2024-04-25T09:23:52.727 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:47d29e85:::benchmark_data_smithi102_461693_object89:head# 2024-04-25T09:23:52.727 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:485eaaa8:::benchmark_data_smithi102_461693_object66:head# 2024-04-25T09:23:52.728 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:49105468:::benchmark_data_smithi102_461693_object76:head# 2024-04-25T09:23:52.728 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:49d9b3e4:::benchmark_data_smithi102_461693_object268:head# 2024-04-25T09:23:52.729 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4a341421:::benchmark_data_smithi102_461693_object288:head# 2024-04-25T09:23:52.730 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4b1ffa35:::benchmark_data_smithi102_461693_object286:head# 2024-04-25T09:23:52.730 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4d5b2900:::benchmark_data_smithi102_461693_object23:head# 2024-04-25T09:23:52.731 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4d6d7ce3:::benchmark_data_smithi102_461693_object122:head# 2024-04-25T09:23:52.732 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4e5a5bae:::benchmark_data_smithi102_461693_object214:head# 2024-04-25T09:23:52.733 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4ee4fe84:::benchmark_data_smithi102_461693_object63:head# 2024-04-25T09:23:52.733 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4f4aa74e:::benchmark_data_smithi102_461693_object146:head# 2024-04-25T09:23:52.734 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4f7e5631:::benchmark_data_smithi102_461693_object251:head# 2024-04-25T09:23:52.734 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4f9e0719:::benchmark_data_smithi102_461693_object222:head# 2024-04-25T09:23:52.735 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:52c86389:::benchmark_data_smithi102_461693_object223:head# 2024-04-25T09:23:52.735 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5308d886:::benchmark_data_smithi102_461693_object45:head# 2024-04-25T09:23:52.736 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5403a8e1:::benchmark_data_smithi102_461693_object276:head# 2024-04-25T09:23:52.737 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:547cc0c2:::benchmark_data_smithi102_461693_object169:head# 2024-04-25T09:23:52.738 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:54f5df8b:::benchmark_data_smithi102_461693_object98:head# 2024-04-25T09:23:52.738 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5503c0c7:::benchmark_data_smithi102_461693_object133:head# 2024-04-25T09:23:52.739 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:557ea668:::benchmark_data_smithi102_461693_object65:head# 2024-04-25T09:23:52.740 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:57f2a07b:::benchmark_data_smithi102_461693_object293:head# 2024-04-25T09:23:52.740 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:580d47d3:::benchmark_data_smithi102_461693_object10:head# 2024-04-25T09:23:52.742 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:58a91fd0:::benchmark_data_smithi102_461693_object292:head# 2024-04-25T09:23:52.742 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5a9a4702:::benchmark_data_smithi102_461693_object30:head# 2024-04-25T09:23:52.743 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5b7a7287:::benchmark_data_smithi102_461693_object178:head# 2024-04-25T09:23:52.744 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5c075ec8:::benchmark_data_smithi102_461693_object91:head# 2024-04-25T09:23:52.744 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5d5f48ea:::benchmark_data_smithi102_461693_object302:head# 2024-04-25T09:23:52.745 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5d8f2267:::benchmark_data_smithi102_461693_object247:head# 2024-04-25T09:23:52.745 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5ec1726b:::benchmark_data_smithi102_461693_object126:head# 2024-04-25T09:23:52.746 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5fe7afcc:::benchmark_data_smithi102_461693_object238:head# 2024-04-25T09:23:52.747 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:60ff7266:::benchmark_data_smithi102_461693_object42:head# 2024-04-25T09:23:52.748 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:61051e5f:::benchmark_data_smithi102_461693_object128:head# 2024-04-25T09:23:52.748 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:638903a9:::benchmark_data_smithi102_461693_object271:head# 2024-04-25T09:23:52.749 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:64392431:::benchmark_data_smithi102_461693_object316:head# 2024-04-25T09:23:52.750 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6452bd93:::benchmark_data_smithi102_461693_object78:head# 2024-04-25T09:23:52.750 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6469f64c:::benchmark_data_smithi102_461693_object246:head# 2024-04-25T09:23:52.751 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:64ce6296:::benchmark_data_smithi102_461693_object100:head# 2024-04-25T09:23:52.752 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6590eadb:::benchmark_data_smithi102_461693_object159:head# 2024-04-25T09:23:52.752 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:665d634c:::benchmark_data_smithi102_461693_object90:head# 2024-04-25T09:23:52.753 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6748bc13:::benchmark_data_smithi102_461693_object36:head# 2024-04-25T09:23:52.754 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6785c648:::benchmark_data_smithi102_461693_object1:head# 2024-04-25T09:23:52.754 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6823a015:::benchmark_data_smithi102_461693_object254:head# 2024-04-25T09:23:52.755 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:68498f01:::benchmark_data_smithi102_461693_object280:head# 2024-04-25T09:23:52.755 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:68811066:::benchmark_data_smithi102_461693_object163:head# 2024-04-25T09:23:52.756 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:68b09bf8:::benchmark_data_smithi102_461693_object224:head# 2024-04-25T09:23:52.757 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:68c982f8:::benchmark_data_smithi102_461693_object97:head# 2024-04-25T09:23:52.757 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:69b24405:::benchmark_data_smithi102_461693_object294:head# 2024-04-25T09:23:52.758 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:69e50a6f:::benchmark_data_smithi102_461693_object143:head# 2024-04-25T09:23:52.759 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6a328a99:::benchmark_data_smithi102_461693_object120:head# 2024-04-25T09:23:52.759 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6aea4f95:::benchmark_data_smithi102_461693_object71:head# 2024-04-25T09:23:52.760 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6b81e8fb:::benchmark_data_smithi102_461693_object140:head# 2024-04-25T09:23:52.760 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6ba5d254:::benchmark_data_smithi102_461693_object68:head# 2024-04-25T09:23:52.760 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6bf4899e:::benchmark_data_smithi102_461693_object116:head# 2024-04-25T09:23:52.760 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6d1b960b:::benchmark_data_smithi102_461693_object35:head# 2024-04-25T09:23:52.761 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6df62e7a:::benchmark_data_smithi102_461693_object56:head# 2024-04-25T09:23:52.762 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6fd57c98:::benchmark_data_smithi102_461693_object162:head# 2024-04-25T09:23:52.762 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:70d697fa:::benchmark_data_smithi102_461693_object297:head# 2024-04-25T09:23:52.763 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:72074391:::benchmark_data_smithi102_461693_object102:head# 2024-04-25T09:23:52.764 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:74228ab3:::benchmark_data_smithi102_461693_object313:head# 2024-04-25T09:23:52.764 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:745a30ad:::benchmark_data_smithi102_461693_object75:head# 2024-04-25T09:23:52.765 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:762df197:::benchmark_data_smithi102_461693_object241:head# 2024-04-25T09:23:52.766 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7744fca5:::benchmark_data_smithi102_461693_object318:head# 2024-04-25T09:23:52.766 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:77ffafc4:::benchmark_data_smithi102_461693_object24:head# 2024-04-25T09:23:52.767 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:78013faf:::benchmark_data_smithi102_461693_object175:head# 2024-04-25T09:23:52.767 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7902734b:::benchmark_data_smithi102_461693_object135:head# 2024-04-25T09:23:52.768 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7958dd60:::benchmark_data_smithi102_461693_object190:head# 2024-04-25T09:23:52.769 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7a196166:::benchmark_data_smithi102_461693_object309:head# 2024-04-25T09:23:52.770 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7a932c37:::benchmark_data_smithi102_461693_object179:head# 2024-04-25T09:23:52.770 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7af1be98:::benchmark_data_smithi102_461693_object298:head# 2024-04-25T09:23:52.771 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7b48691d:::benchmark_data_smithi102_461693_object79:head# 2024-04-25T09:23:52.772 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7cc3111f:::benchmark_data_smithi102_461693_object44:head# 2024-04-25T09:23:52.772 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7cf5ad47:::benchmark_data_smithi102_461693_object121:head# 2024-04-25T09:23:52.773 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7d1cce5a:::benchmark_data_smithi102_461693_object206:head# 2024-04-25T09:23:52.774 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7dca102f:::benchmark_data_smithi102_461693_object202:head# 2024-04-25T09:23:52.774 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7df4b946:::benchmark_data_smithi102_461693_object245:head# 2024-04-25T09:23:52.775 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7e06601e:::benchmark_data_smithi102_461693_object252:head# 2024-04-25T09:23:52.775 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7f86f02e:::benchmark_data_smithi102_461693_object138:head# 2024-04-25T09:23:52.775 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7fb155ff:::benchmark_data_smithi102_461693_object188:head# 2024-04-25T09:23:52.776 INFO:tasks.workunit.client.0.smithi102.stderr:Export successful 2024-04-25T09:23:53.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:102: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:23:53.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:23:53.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:23:53.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:23:53.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:23:53.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:23:53.070 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:23:53.070 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:23:53.070 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:23:53.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:23:53.072 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:23:53.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:23:53.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:23:53.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:23:53.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:23:53.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:23:53.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:23:53.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:23:53.107 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:53.120+0000 7fc1d13a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:53.110 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:53.120+0000 7fc1d13a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:53.112 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:53.124+0000 7fc1d13a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:53.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:23:53.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:23:53.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:23:53.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:23:53.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:23:53.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:53.517 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:23:53.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:23:53.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:53.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:53.692 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:53.704+0000 7fc1d13a57c0 -1 Falling back to public interface 2024-04-25T09:23:53.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:54.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:54.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:54.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:23:54.946 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:23:54.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:54.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:55.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:56.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:56.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:56.357 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:23:56.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:23:56.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:56.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:56.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:57.231 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:23:57.244+0000 7fc1d13a57c0 -1 osd.0 10 log_to_monitors true 2024-04-25T09:23:57.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:57.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:57.786 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:23:57.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:23:57.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:57.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:58.259 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/3208710954,v1:127.0.0.1:6803/3208710954] [v2:127.0.0.1:6804/3208710954,v1:127.0.0.1:6805/3208710954] exists,up 150f6751-a5c0-42ad-b859-7e7a4382302c 2024-04-25T09:23:58.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:23:58.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:23:58.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:23:58.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:104: TEST_import_after_merge_and_gap: ceph osd pool set foo pg_num 1 2024-04-25T09:23:59.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:105: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:24:04.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: ceph daemon osd.0 perf dump 2024-04-25T09:24:04.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: jq .osd.numpg 2024-04-25T09:24:04.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: grep 2 2024-04-25T09:24:04.184 INFO:tasks.workunit.client.0.smithi102.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-25T09:24:04.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:107: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:24:04.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:24:04.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:24:04.186 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:24:04.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:24:04.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:24:04.187 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:24:04.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:24:04.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:24:04.187 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:24:04.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:24:04.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:24:04.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:24:04.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:24:04.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:24:04.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:24:04.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:24:04.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:24:04.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:04.707 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:24:04.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574852 2024-04-25T09:24:04.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574852 2024-04-25T09:24:04.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-55834574852' 2024-04-25T09:24:04.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:04.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-55834574852 2024-04-25T09:24:04.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:04.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:24:04.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-55834574852 2024-04-25T09:24:04.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:04.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574852 2024-04-25T09:24:04.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-04-25T09:24:04.882 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 55834574852 2024-04-25T09:24:04.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:24:05.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574850 -lt 55834574852 2024-04-25T09:24:05.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:24:06.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:24:06.284 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:24:06.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574852 -lt 55834574852 2024-04-25T09:24:06.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:24:06.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:06.774 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:07.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:24:07.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:24:07.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:24:07.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:24:07.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:24:07.305 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:24:07.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:24:07.306 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:24:07.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:24:07.693 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:24:07.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:07.694 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:24:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:24:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:24:08.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:110: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:24:08.220 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:24:08.220 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:24:08.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:24:08.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:24:08.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:24:08.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:24:08.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:111: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:24:09.786 INFO:tasks.workunit.client.0.smithi102.stdout: marking collection for removal 2024-04-25T09:24:09.787 INFO:tasks.workunit.client.0.smithi102.stdout:setting '_remove' omap key 2024-04-25T09:24:09.787 INFO:tasks.workunit.client.0.smithi102.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:24:10.345 INFO:tasks.workunit.client.0.smithi102.stdout:Remove successful 2024-04-25T09:24:10.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:113: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:24:11.793 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e21 pg_nums {1={7=2,20=1}} deleted_pools ) 2024-04-25T09:24:11.793 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.1 2024-04-25T09:24:11.793 INFO:tasks.workunit.client.0.smithi102.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:24:11.793 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:80decefa:::benchmark_data_smithi102_461693_object215:head# 2024-04-25T09:24:11.793 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:11.872 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:82504851:::benchmark_data_smithi102_461693_object59:head# 2024-04-25T09:24:11.873 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:11.914 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:82bc124c:::benchmark_data_smithi102_461693_object242:head# 2024-04-25T09:24:11.914 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:11.956 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:883cb801:::benchmark_data_smithi102_461693_object150:head# 2024-04-25T09:24:11.956 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:11.998 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:889e9aa5:::benchmark_data_smithi102_461693_object250:head# 2024-04-25T09:24:11.998 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.040 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:890d9aaf:::benchmark_data_smithi102_461693_object96:head# 2024-04-25T09:24:12.040 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.081 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8a093910:::benchmark_data_smithi102_461693_object15:head# 2024-04-25T09:24:12.082 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.148 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8aedfdc0:::benchmark_data_smithi102_461693_object193:head# 2024-04-25T09:24:12.148 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.215 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8afeae38:::benchmark_data_smithi102_461693_object221:head# 2024-04-25T09:24:12.215 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.291 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8b078005:::benchmark_data_smithi102_461693_object60:head# 2024-04-25T09:24:12.291 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.333 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8b4ee6c4:::benchmark_data_smithi102_461693_object166:head# 2024-04-25T09:24:12.333 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.374 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8ca6a195:::benchmark_data_smithi102_461693_object172:head# 2024-04-25T09:24:12.375 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.408 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8cdff5e0:::benchmark_data_smithi102_461693_object17:head# 2024-04-25T09:24:12.408 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.441 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8d571cff:::benchmark_data_smithi102_461693_object255:head# 2024-04-25T09:24:12.442 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.475 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8eeaf510:::benchmark_data_smithi102_461693_object22:head# 2024-04-25T09:24:12.475 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.508 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8f3dbf2e:::benchmark_data_smithi102_461693_object110:head# 2024-04-25T09:24:12.508 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.542 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:91d0aef9:::benchmark_data_smithi102_461693_object132:head# 2024-04-25T09:24:12.542 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.575 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:93bbd15e:::benchmark_data_smithi102_461693_object311:head# 2024-04-25T09:24:12.575 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.609 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:942bc514:::benchmark_data_smithi102_461693_object77:head# 2024-04-25T09:24:12.609 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.642 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:94d0ab29:::benchmark_data_smithi102_461693_object285:head# 2024-04-25T09:24:12.642 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.675 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:94ec3b32:::benchmark_data_smithi102_461693_object72:head# 2024-04-25T09:24:12.675 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.709 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:95092f81:::benchmark_data_smithi102_461693_object220:head# 2024-04-25T09:24:12.709 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.742 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:95b06a69:::benchmark_data_smithi102_461693_object67:head# 2024-04-25T09:24:12.743 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.776 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:96ade7c2:::benchmark_data_smithi102_461693_object115:head# 2024-04-25T09:24:12.776 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.809 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:971cae83:::benchmark_data_smithi102_461693_object196:head# 2024-04-25T09:24:12.809 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.843 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:978b03f9:::benchmark_data_smithi102_461693_object236:head# 2024-04-25T09:24:12.843 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.876 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:98701baa:::benchmark_data_smithi102_461693_object55:head# 2024-04-25T09:24:12.876 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.910 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:98e4a738:::benchmark_data_smithi102_461693_object127:head# 2024-04-25T09:24:12.910 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.943 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9aa9891c:::benchmark_data_smithi102_461693_object5:head# 2024-04-25T09:24:12.943 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:12.977 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9afe2884:::benchmark_data_smithi102_461693_object273:head# 2024-04-25T09:24:12.977 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.010 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9bd5d2e8:::benchmark_data_smithi102_461693_object265:head# 2024-04-25T09:24:13.011 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.044 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9bf31ddf:::benchmark_data_smithi102_461693_object83:head# 2024-04-25T09:24:13.044 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.077 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9c9b760d:::benchmark_data_smithi102_461693_object291:head# 2024-04-25T09:24:13.077 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.111 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9c9c7e9b:::benchmark_data_smithi102_461693_object70:head# 2024-04-25T09:24:13.111 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.152 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9cc82558:::benchmark_data_smithi102_461693_object108:head# 2024-04-25T09:24:13.152 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.186 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9e60200b:::benchmark_data_smithi102_461693_object26:head# 2024-04-25T09:24:13.186 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.219 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9ebdc7a5:::benchmark_data_smithi102_461693_object18:head# 2024-04-25T09:24:13.219 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.253 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9eccdf33:::benchmark_data_smithi102_461693_object131:head# 2024-04-25T09:24:13.253 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.286 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9fcd1944:::benchmark_data_smithi102_461693_object198:head# 2024-04-25T09:24:13.286 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.328 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9fcee8c9:::benchmark_data_smithi102_461693_object263:head# 2024-04-25T09:24:13.328 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.370 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a109394d:::benchmark_data_smithi102_461693_object49:head# 2024-04-25T09:24:13.370 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.411 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a284d2b1:::benchmark_data_smithi102_461693_object40:head# 2024-04-25T09:24:13.412 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.453 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a2ff1fbf:::benchmark_data_smithi102_461693_object52:head# 2024-04-25T09:24:13.454 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.495 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a32db852:::benchmark_data_smithi102_461693_object232:head# 2024-04-25T09:24:13.495 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.537 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a4f4d3a4:::benchmark_data_smithi102_461693_object261:head# 2024-04-25T09:24:13.537 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.579 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a7538180:::benchmark_data_smithi102_461693_object308:head# 2024-04-25T09:24:13.579 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.646 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aa0c2ca6:::benchmark_data_smithi102_461693_object213:head# 2024-04-25T09:24:13.646 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.696 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aa2b501d:::benchmark_data_smithi102_461693_object218:head# 2024-04-25T09:24:13.696 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.738 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aae249da:::benchmark_data_smithi102_461693_object182:head# 2024-04-25T09:24:13.738 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.780 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac080002:::benchmark_data_smithi102_461693_object13:head# 2024-04-25T09:24:13.780 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.821 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac64182c:::benchmark_data_smithi102_461693_object295:head# 2024-04-25T09:24:13.821 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.863 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac96a035:::benchmark_data_smithi102_461693_object46:head# 2024-04-25T09:24:13.863 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.905 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:acd4c3d0:::benchmark_data_smithi102_461693_object226:head# 2024-04-25T09:24:13.905 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.947 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ad92f4ab:::benchmark_data_smithi102_461693_object303:head# 2024-04-25T09:24:13.947 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:13.989 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:adef340d:::benchmark_data_smithi102_461693_object61:head# 2024-04-25T09:24:14.023 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.030 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b1d35c85:::benchmark_data_smithi102_461693_object200:head# 2024-04-25T09:24:14.031 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.072 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b1d73141:::benchmark_data_smithi102_461693_object112:head# 2024-04-25T09:24:14.072 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.122 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b4852218:::benchmark_data_smithi102_461693_object277:head# 2024-04-25T09:24:14.122 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.207 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b56c2919:::benchmark_data_smithi102_461693_object270:head# 2024-04-25T09:24:14.207 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.290 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b56e0ab2:::benchmark_data_smithi102_461693_object231:head# 2024-04-25T09:24:14.290 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.332 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b5c35fa0:::benchmark_data_smithi102_461693_object130:head# 2024-04-25T09:24:14.332 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.374 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b7e1813e:::benchmark_data_smithi102_461693_object185:head# 2024-04-25T09:24:14.374 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.416 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:24:14.416 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.457 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b8772720:::benchmark_data_smithi102_461693_object289:head# 2024-04-25T09:24:14.457 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.499 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bba685cf:::benchmark_data_smithi102_461693_object155:head# 2024-04-25T09:24:14.499 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.574 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc36f44b:::benchmark_data_smithi102_461693_object296:head# 2024-04-25T09:24:14.574 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.616 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc9448f0:::benchmark_data_smithi102_461693_object82:head# 2024-04-25T09:24:14.616 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.658 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bd20be31:::benchmark_data_smithi102_461693_object93:head# 2024-04-25T09:24:14.658 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.699 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bda0949f:::benchmark_data_smithi102_461693_object181:head# 2024-04-25T09:24:14.700 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.741 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bde44e2f:::benchmark_data_smithi102_461693_object95:head# 2024-04-25T09:24:14.742 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.783 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:be6d30be:::benchmark_data_smithi102_461693_object207:head# 2024-04-25T09:24:14.783 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.825 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bf7d046b:::benchmark_data_smithi102_461693_object278:head# 2024-04-25T09:24:14.825 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.867 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c1eaa85c:::benchmark_data_smithi102_461693_object137:head# 2024-04-25T09:24:14.867 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.908 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c37f1115:::benchmark_data_smithi102_461693_object310:head# 2024-04-25T09:24:14.909 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.950 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c50db599:::benchmark_data_smithi102_461693_object176:head# 2024-04-25T09:24:14.951 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:14.984 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c53ef393:::benchmark_data_smithi102_461693_object279:head# 2024-04-25T09:24:14.984 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.018 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c53f44d8:::benchmark_data_smithi102_461693_object239:head# 2024-04-25T09:24:15.018 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.051 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c551c4c0:::benchmark_data_smithi102_461693_object123:head# 2024-04-25T09:24:15.051 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.084 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c55d9b9d:::benchmark_data_smithi102_461693_object237:head# 2024-04-25T09:24:15.084 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.118 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c6ad85a0:::benchmark_data_smithi102_461693_object41:head# 2024-04-25T09:24:15.118 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.151 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c6d379b1:::benchmark_data_smithi102_461693_object304:head# 2024-04-25T09:24:15.151 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.185 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c74b1c18:::benchmark_data_smithi102_461693_object113:head# 2024-04-25T09:24:15.185 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.218 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c7f88b4e:::benchmark_data_smithi102_461693_object284:head# 2024-04-25T09:24:15.219 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.251 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c88cd082:::benchmark_data_smithi102_461693_object259:head# 2024-04-25T09:24:15.252 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.285 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c8e2e872:::benchmark_data_smithi102_461693_object39:head# 2024-04-25T09:24:15.285 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.318 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c93a1b98:::benchmark_data_smithi102_461693_object142:head# 2024-04-25T09:24:15.318 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.352 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c9c320e5:::benchmark_data_smithi102_461693_object249:head# 2024-04-25T09:24:15.352 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.385 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ca288c06:::benchmark_data_smithi102_461693_object153:head# 2024-04-25T09:24:15.385 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.419 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cba55ca9:::benchmark_data_smithi102_461693_object283:head# 2024-04-25T09:24:15.419 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.452 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cbd30596:::benchmark_data_smithi102_461693_object145:head# 2024-04-25T09:24:15.453 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.486 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cc3765a4:::benchmark_data_smithi102_461693_object0:head# 2024-04-25T09:24:15.486 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.519 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ce561cbb:::benchmark_data_smithi102_461693_object244:head# 2024-04-25T09:24:15.520 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.552 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cec1e000:::benchmark_data_smithi102_461693_object171:head# 2024-04-25T09:24:15.553 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.596 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d07784f4:::benchmark_data_smithi102_461693_object107:head# 2024-04-25T09:24:15.597 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.678 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d131843b:::benchmark_data_smithi102_461693_object119:head# 2024-04-25T09:24:15.678 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.711 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d1f2e35b:::benchmark_data_smithi102_461693_object257:head# 2024-04-25T09:24:15.712 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.745 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d2bb2fe0:::benchmark_data_smithi102_461693_object197:head# 2024-04-25T09:24:15.745 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.778 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d2f5066b:::benchmark_data_smithi102_461693_object307:head# 2024-04-25T09:24:15.778 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.812 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d77f7f03:::benchmark_data_smithi102_461693_object170:head# 2024-04-25T09:24:15.812 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.845 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d95e4094:::benchmark_data_smithi102_461693_object105:head# 2024-04-25T09:24:15.846 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.879 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:da23c414:::benchmark_data_smithi102_461693_object57:head# 2024-04-25T09:24:15.879 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.912 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:da5b1f76:::benchmark_data_smithi102_461693_object73:head# 2024-04-25T09:24:15.913 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.946 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:da5b6865:::benchmark_data_smithi102_461693_object290:head# 2024-04-25T09:24:15.946 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:15.988 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:db2f70fa:::benchmark_data_smithi102_461693_object230:head# 2024-04-25T09:24:15.988 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.029 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dbf5b9d1:::benchmark_data_smithi102_461693_object312:head# 2024-04-25T09:24:16.030 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.071 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:de452fc5:::benchmark_data_smithi102_461693_object167:head# 2024-04-25T09:24:16.071 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.122 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e195d889:::benchmark_data_smithi102_461693_object129:head# 2024-04-25T09:24:16.122 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.197 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e2323390:::benchmark_data_smithi102_461693_object33:head# 2024-04-25T09:24:16.197 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.273 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e2b7382e:::benchmark_data_smithi102_461693_object210:head# 2024-04-25T09:24:16.273 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.314 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e2bbe377:::benchmark_data_smithi102_461693_object114:head# 2024-04-25T09:24:16.315 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.356 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e301fcee:::benchmark_data_smithi102_461693_object62:head# 2024-04-25T09:24:16.357 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.398 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e302dfe1:::benchmark_data_smithi102_461693_object227:head# 2024-04-25T09:24:16.398 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.440 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e4324c30:::benchmark_data_smithi102_461693_object158:head# 2024-04-25T09:24:16.440 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.481 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e447aef9:::benchmark_data_smithi102_461693_object180:head# 2024-04-25T09:24:16.481 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.523 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e4558303:::benchmark_data_smithi102_461693_object4:head# 2024-04-25T09:24:16.523 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.565 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e5053873:::benchmark_data_smithi102_461693_object81:head# 2024-04-25T09:24:16.565 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.607 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e5210074:::benchmark_data_smithi102_461693_object3:head# 2024-04-25T09:24:16.607 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.648 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e7382a9c:::benchmark_data_smithi102_461693_object299:head# 2024-04-25T09:24:16.649 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.690 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e819bcc3:::benchmark_data_smithi102_461693_object84:head# 2024-04-25T09:24:16.690 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.732 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e8d6bf29:::benchmark_data_smithi102_461693_object101:head# 2024-04-25T09:24:16.732 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.774 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ea24f1cb:::benchmark_data_smithi102_461693_object186:head# 2024-04-25T09:24:16.774 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.816 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ec9bc09f:::benchmark_data_smithi102_461693_object229:head# 2024-04-25T09:24:16.816 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.858 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ecca2721:::benchmark_data_smithi102_461693_object134:head# 2024-04-25T09:24:16.858 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.899 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ecdd65cb:::benchmark_data_smithi102_461693_object269:head# 2024-04-25T09:24:16.899 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.958 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:eed87a51:::benchmark_data_smithi102_461693_object191:head# 2024-04-25T09:24:16.958 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:16.999 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ef8637ba:::benchmark_data_smithi102_461693_object151:head# 2024-04-25T09:24:17.000 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.041 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f02b8d22:::benchmark_data_smithi102_461693_object168:head# 2024-04-25T09:24:17.042 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.083 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f06a39d9:::benchmark_data_smithi102_461693_object275:head# 2024-04-25T09:24:17.083 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.125 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f136e2f9:::benchmark_data_smithi102_461693_object267:head# 2024-04-25T09:24:17.125 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.175 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f2467763:::benchmark_data_smithi102_461693_object156:head# 2024-04-25T09:24:17.175 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.217 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f3685b12:::benchmark_data_smithi102_461693_object212:head# 2024-04-25T09:24:17.217 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.259 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f39374a5:::benchmark_data_smithi102_461693_object184:head# 2024-04-25T09:24:17.259 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.300 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f5c19b84:::benchmark_data_smithi102_461693_object225:head# 2024-04-25T09:24:17.301 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.342 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f69416ea:::benchmark_data_smithi102_461693_object235:head# 2024-04-25T09:24:17.342 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.384 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f7295629:::benchmark_data_smithi102_461693_object87:head# 2024-04-25T09:24:17.384 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.426 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f90d383f:::benchmark_data_smithi102_461693_object203:head# 2024-04-25T09:24:17.426 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.468 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f939e8fc:::benchmark_data_smithi102_461693_object48:head# 2024-04-25T09:24:17.468 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.509 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f993a43e:::benchmark_data_smithi102_461693_object99:head# 2024-04-25T09:24:17.510 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.551 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f9fdbd9c:::benchmark_data_smithi102_461693_object86:head# 2024-04-25T09:24:17.551 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.642 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fa2a1c9b:::benchmark_data_smithi102_461693_object47:head# 2024-04-25T09:24:17.643 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.735 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fb939772:::benchmark_data_smithi102_461693_object9:head# 2024-04-25T09:24:17.735 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.769 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fca3284e:::benchmark_data_smithi102_461693_object187:head# 2024-04-25T09:24:17.769 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.802 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fe98a8ca:::benchmark_data_smithi102_461693_object258:head# 2024-04-25T09:24:17.802 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.835 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ff7040e0:::benchmark_data_smithi102_461693_object281:head# 2024-04-25T09:24:17.836 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.869 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ffab2f9e:::benchmark_data_smithi102_461693_object154:head# 2024-04-25T09:24:17.869 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:17.902 INFO:tasks.workunit.client.0.smithi102.stdout:write_pg epoch 10 info 1.1( v 10'145 (0'0,10'145] local-lis/les=7/8 n=145 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:24:17.944 INFO:tasks.workunit.client.0.smithi102.stdout:Import successful 2024-04-25T09:24:18.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:114: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:24:19.841 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e21 pg_nums {1={7=2,20=1}} deleted_pools ) 2024-04-25T09:24:19.842 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.0 2024-04-25T09:24:19.842 INFO:tasks.workunit.client.0.smithi102.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:24:19.842 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:000b3b92:::benchmark_data_smithi102_461693_object262:head# 2024-04-25T09:24:19.842 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:19.933 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:00374c2e:::benchmark_data_smithi102_461693_object216:head# 2024-04-25T09:24:19.933 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:19.975 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0301c80c:::benchmark_data_smithi102_461693_object11:head# 2024-04-25T09:24:19.975 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.016 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:036651e6:::benchmark_data_smithi102_461693_object111:head# 2024-04-25T09:24:20.017 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.058 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0402916e:::benchmark_data_smithi102_461693_object183:head# 2024-04-25T09:24:20.058 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.114 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:046b3940:::benchmark_data_smithi102_461693_object104:head# 2024-04-25T09:24:20.114 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.209 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0540312f:::benchmark_data_smithi102_461693_object69:head# 2024-04-25T09:24:20.209 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.284 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:054ee2e1:::benchmark_data_smithi102_461693_object300:head# 2024-04-25T09:24:20.284 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.318 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:069b73e0:::benchmark_data_smithi102_461693_object195:head# 2024-04-25T09:24:20.318 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.376 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:07526ebf:::benchmark_data_smithi102_461693_object248:head# 2024-04-25T09:24:20.377 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.426 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:08aeff1a:::benchmark_data_smithi102_461693_object43:head# 2024-04-25T09:24:20.427 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.476 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:09569df1:::benchmark_data_smithi102_461693_object305:head# 2024-04-25T09:24:20.477 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.527 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:099020c4:::benchmark_data_smithi102_461693_object152:head# 2024-04-25T09:24:20.527 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.577 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0a47cb3d:::benchmark_data_smithi102_461693_object266:head# 2024-04-25T09:24:20.577 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.627 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0a7de0f7:::benchmark_data_smithi102_461693_object208:head# 2024-04-25T09:24:20.627 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.702 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0b6a70be:::benchmark_data_smithi102_461693_object85:head# 2024-04-25T09:24:20.702 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.752 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0cc78a03:::benchmark_data_smithi102_461693_object21:head# 2024-04-25T09:24:20.752 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.794 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0cd9cd5d:::benchmark_data_smithi102_461693_object160:head# 2024-04-25T09:24:20.794 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.836 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0deb35d3:::benchmark_data_smithi102_461693_object217:head# 2024-04-25T09:24:20.836 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.877 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0e8b2dd0:::benchmark_data_smithi102_461693_object148:head# 2024-04-25T09:24:20.877 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.919 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0f24b073:::benchmark_data_smithi102_461693_object317:head# 2024-04-25T09:24:20.919 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:20.961 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:10af48fb:::benchmark_data_smithi102_461693_object240:head# 2024-04-25T09:24:20.961 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.011 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:11692bdb:::benchmark_data_smithi102_461693_object27:head# 2024-04-25T09:24:21.011 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.061 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:128e9745:::benchmark_data_smithi102_461693_object314:head# 2024-04-25T09:24:21.061 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.111 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:132fbbae:::benchmark_data_smithi102_461693_object92:head# 2024-04-25T09:24:21.111 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.153 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:14a6a0cd:::benchmark_data_smithi102_461693_object88:head# 2024-04-25T09:24:21.153 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.195 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1512b06b:::benchmark_data_smithi102_461693_object272:head# 2024-04-25T09:24:21.195 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.237 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1629e659:::benchmark_data_smithi102_461693_object256:head# 2024-04-25T09:24:21.237 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.279 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:16f7bd8c:::benchmark_data_smithi102_461693_object64:head# 2024-04-25T09:24:21.279 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.320 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1786312c:::benchmark_data_smithi102_461693_object37:head# 2024-04-25T09:24:21.320 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.362 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:19fa802a:::benchmark_data_smithi102_461693_object177:head# 2024-04-25T09:24:21.362 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.404 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1d2fbc72:::benchmark_data_smithi102_461693_object253:head# 2024-04-25T09:24:21.404 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.445 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1dbe156c:::benchmark_data_smithi102_461693_object301:head# 2024-04-25T09:24:21.445 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.487 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1e43a575:::benchmark_data_smithi102_461693_object109:head# 2024-04-25T09:24:21.488 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.529 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1e98b968:::benchmark_data_smithi102_461693_object8:head# 2024-04-25T09:24:21.529 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.571 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1e9976c2:::benchmark_data_smithi102_461693_object31:head# 2024-04-25T09:24:21.571 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.663 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:20ac13d7:::benchmark_data_smithi102_461693_object319:head# 2024-04-25T09:24:21.664 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.730 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2279d77a:::benchmark_data_smithi102_461693_object157:head# 2024-04-25T09:24:21.730 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.772 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:22b40e23:::benchmark_data_smithi102_461693_object233:head# 2024-04-25T09:24:21.772 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.814 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:23111692:::benchmark_data_smithi102_461693_object14:head# 2024-04-25T09:24:21.814 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.855 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:23426916:::benchmark_data_smithi102_461693_object164:head# 2024-04-25T09:24:21.856 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.897 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:238a3ae3:::benchmark_data_smithi102_461693_object211:head# 2024-04-25T09:24:21.898 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.939 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:239e27b3:::benchmark_data_smithi102_461693_object219:head# 2024-04-25T09:24:21.939 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:21.981 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:23b4b493:::benchmark_data_smithi102_461693_object192:head# 2024-04-25T09:24:21.981 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.023 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:24e58257:::benchmark_data_smithi102_461693_object141:head# 2024-04-25T09:24:22.023 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.064 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:24e6a109:::benchmark_data_smithi102_461693_object106:head# 2024-04-25T09:24:22.064 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.113 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:25997359:::benchmark_data_smithi102_461693_object264:head# 2024-04-25T09:24:22.114 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.215 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:25b16926:::benchmark_data_smithi102_461693_object29:head# 2024-04-25T09:24:22.215 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.299 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:26824afa:::benchmark_data_smithi102_461693_object94:head# 2024-04-25T09:24:22.299 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.341 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:26a55470:::benchmark_data_smithi102_461693_object274:head# 2024-04-25T09:24:22.341 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.383 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:272eb2c9:::benchmark_data_smithi102_461693_object149:head# 2024-04-25T09:24:22.383 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.424 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:273627fb:::benchmark_data_smithi102_461693_object125:head# 2024-04-25T09:24:22.424 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.466 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:273b3e12:::benchmark_data_smithi102_461693_object161:head# 2024-04-25T09:24:22.466 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.508 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:275d130a:::benchmark_data_smithi102_461693_object20:head# 2024-04-25T09:24:22.508 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.550 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:28694220:::benchmark_data_smithi102_461693_object136:head# 2024-04-25T09:24:22.550 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.591 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2a0bbe58:::benchmark_data_smithi102_461693_object199:head# 2024-04-25T09:24:22.592 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.633 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2afc2f48:::benchmark_data_smithi102_461693_object209:head# 2024-04-25T09:24:22.633 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.675 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2bbb9370:::benchmark_data_smithi102_461693_object287:head# 2024-04-25T09:24:22.675 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.717 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2bd520cd:::benchmark_data_smithi102_461693_object19:head# 2024-04-25T09:24:22.717 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.759 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2be202af:::benchmark_data_smithi102_461693_object234:head# 2024-04-25T09:24:22.759 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.827 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2c0b4651:::benchmark_data_smithi102_461693_object74:head# 2024-04-25T09:24:22.827 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.892 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2c9a727c:::benchmark_data_smithi102_461693_object165:head# 2024-04-25T09:24:22.893 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.934 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2d3d0890:::benchmark_data_smithi102_461693_object50:head# 2024-04-25T09:24:22.935 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:22.976 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2d3fef0a:::benchmark_data_smithi102_461693_object173:head# 2024-04-25T09:24:22.976 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.018 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2d4d8992:::benchmark_data_smithi102_461693_object53:head# 2024-04-25T09:24:23.018 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.060 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2ee095c4:::benchmark_data_smithi102_461693_object58:head# 2024-04-25T09:24:23.060 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.101 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2fd07b30:::benchmark_data_smithi102_461693_object194:head# 2024-04-25T09:24:23.102 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.143 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2fee0198:::benchmark_data_smithi102_461693_object2:head# 2024-04-25T09:24:23.144 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.185 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:34e9f16c:::benchmark_data_smithi102_461693_object260:head# 2024-04-25T09:24:23.185 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.227 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:355453a0:::benchmark_data_smithi102_461693_object118:head# 2024-04-25T09:24:23.227 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.268 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:363cd543:::benchmark_data_smithi102_461693_object34:head# 2024-04-25T09:24:23.269 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.302 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:36551f3c:::benchmark_data_smithi102_461693_object51:head# 2024-04-25T09:24:23.302 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.335 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3693711a:::benchmark_data_smithi102_461693_object306:head# 2024-04-25T09:24:23.335 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.369 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:37225bcc:::benchmark_data_smithi102_461693_object117:head# 2024-04-25T09:24:23.369 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.411 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:37c0cca3:::benchmark_data_smithi102_461693_object282:head# 2024-04-25T09:24:23.411 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.461 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:389b2f68:::benchmark_data_smithi102_461693_object54:head# 2024-04-25T09:24:23.461 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.511 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:38f5ec38:::benchmark_data_smithi102_461693_object12:head# 2024-04-25T09:24:23.511 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.561 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:39a2e424:::benchmark_data_smithi102_461693_object16:head# 2024-04-25T09:24:23.561 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.670 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3b72c6d5:::benchmark_data_smithi102_461693_object28:head# 2024-04-25T09:24:23.670 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.720 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3bca4278:::benchmark_data_smithi102_461693_object6:head# 2024-04-25T09:24:23.720 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.796 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3c78f5d8:::benchmark_data_smithi102_461693_object124:head# 2024-04-25T09:24:23.796 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.837 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3ccb0f2b:::benchmark_data_smithi102_461693_object144:head# 2024-04-25T09:24:23.837 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.879 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3d133806:::benchmark_data_smithi102_461693_object243:head# 2024-04-25T09:24:23.879 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.920 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3d2ad51e:::benchmark_data_smithi102_461693_object38:head# 2024-04-25T09:24:23.921 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:23.970 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3d611f0a:::benchmark_data_smithi102_461693_object32:head# 2024-04-25T09:24:23.971 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.021 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3f6d6ad9:::benchmark_data_smithi102_461693_object204:head# 2024-04-25T09:24:24.021 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.063 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3f7d2427:::benchmark_data_smithi102_461693_object205:head# 2024-04-25T09:24:24.063 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.113 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:40581110:::benchmark_data_smithi102_461693_object147:head# 2024-04-25T09:24:24.113 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.214 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4066b8f9:::benchmark_data_smithi102_461693_object103:head# 2024-04-25T09:24:24.214 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.297 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:407cca9e:::benchmark_data_smithi102_461693_object228:head# 2024-04-25T09:24:24.297 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.339 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:411befe3:::benchmark_data_smithi102_461693_object25:head# 2024-04-25T09:24:24.339 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.381 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:412d23d7:::benchmark_data_smithi102_461693_object174:head# 2024-04-25T09:24:24.381 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.422 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:426f5982:::benchmark_data_smithi102_461693_object7:head# 2024-04-25T09:24:24.423 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.464 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:443e3636:::benchmark_data_smithi102_461693_object201:head# 2024-04-25T09:24:24.464 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.506 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:45dbb99e:::benchmark_data_smithi102_461693_object139:head# 2024-04-25T09:24:24.506 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.548 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:462c230e:::benchmark_data_smithi102_461693_object189:head# 2024-04-25T09:24:24.548 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.590 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:46e420c2:::benchmark_data_smithi102_461693_object315:head# 2024-04-25T09:24:24.590 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.631 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:46ed67bb:::benchmark_data_smithi102_461693_object80:head# 2024-04-25T09:24:24.632 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.673 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:47d29e85:::benchmark_data_smithi102_461693_object89:head# 2024-04-25T09:24:24.673 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.715 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:485eaaa8:::benchmark_data_smithi102_461693_object66:head# 2024-04-25T09:24:24.715 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.757 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:49105468:::benchmark_data_smithi102_461693_object76:head# 2024-04-25T09:24:24.757 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.799 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:49d9b3e4:::benchmark_data_smithi102_461693_object268:head# 2024-04-25T09:24:24.799 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.840 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4a341421:::benchmark_data_smithi102_461693_object288:head# 2024-04-25T09:24:24.841 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.882 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4b1ffa35:::benchmark_data_smithi102_461693_object286:head# 2024-04-25T09:24:24.882 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.924 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4d5b2900:::benchmark_data_smithi102_461693_object23:head# 2024-04-25T09:24:24.924 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:24.966 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4d6d7ce3:::benchmark_data_smithi102_461693_object122:head# 2024-04-25T09:24:24.966 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.007 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4e5a5bae:::benchmark_data_smithi102_461693_object214:head# 2024-04-25T09:24:25.008 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.049 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4ee4fe84:::benchmark_data_smithi102_461693_object63:head# 2024-04-25T09:24:25.049 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.091 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4f4aa74e:::benchmark_data_smithi102_461693_object146:head# 2024-04-25T09:24:25.091 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.133 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4f7e5631:::benchmark_data_smithi102_461693_object251:head# 2024-04-25T09:24:25.133 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4f9e0719:::benchmark_data_smithi102_461693_object222:head# 2024-04-25T09:24:25.175 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.217 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:52c86389:::benchmark_data_smithi102_461693_object223:head# 2024-04-25T09:24:25.217 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.258 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5308d886:::benchmark_data_smithi102_461693_object45:head# 2024-04-25T09:24:25.258 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.300 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5403a8e1:::benchmark_data_smithi102_461693_object276:head# 2024-04-25T09:24:25.300 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.342 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:547cc0c2:::benchmark_data_smithi102_461693_object169:head# 2024-04-25T09:24:25.342 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.384 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:54f5df8b:::benchmark_data_smithi102_461693_object98:head# 2024-04-25T09:24:25.384 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.425 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5503c0c7:::benchmark_data_smithi102_461693_object133:head# 2024-04-25T09:24:25.426 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.467 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:557ea668:::benchmark_data_smithi102_461693_object65:head# 2024-04-25T09:24:25.467 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.509 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:57f2a07b:::benchmark_data_smithi102_461693_object293:head# 2024-04-25T09:24:25.509 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.551 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:580d47d3:::benchmark_data_smithi102_461693_object10:head# 2024-04-25T09:24:25.551 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.605 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:58a91fd0:::benchmark_data_smithi102_461693_object292:head# 2024-04-25T09:24:25.605 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.693 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5a9a4702:::benchmark_data_smithi102_461693_object30:head# 2024-04-25T09:24:25.693 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.735 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5b7a7287:::benchmark_data_smithi102_461693_object178:head# 2024-04-25T09:24:25.735 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.777 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5c075ec8:::benchmark_data_smithi102_461693_object91:head# 2024-04-25T09:24:25.777 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.844 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5d5f48ea:::benchmark_data_smithi102_461693_object302:head# 2024-04-25T09:24:25.844 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.885 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5d8f2267:::benchmark_data_smithi102_461693_object247:head# 2024-04-25T09:24:25.885 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.927 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5ec1726b:::benchmark_data_smithi102_461693_object126:head# 2024-04-25T09:24:25.928 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:25.969 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5fe7afcc:::benchmark_data_smithi102_461693_object238:head# 2024-04-25T09:24:25.969 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.011 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:60ff7266:::benchmark_data_smithi102_461693_object42:head# 2024-04-25T09:24:26.011 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.052 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:61051e5f:::benchmark_data_smithi102_461693_object128:head# 2024-04-25T09:24:26.053 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.094 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:638903a9:::benchmark_data_smithi102_461693_object271:head# 2024-04-25T09:24:26.094 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.170 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:64392431:::benchmark_data_smithi102_461693_object316:head# 2024-04-25T09:24:26.170 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.237 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6452bd93:::benchmark_data_smithi102_461693_object78:head# 2024-04-25T09:24:26.237 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.312 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6469f64c:::benchmark_data_smithi102_461693_object246:head# 2024-04-25T09:24:26.312 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.362 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:64ce6296:::benchmark_data_smithi102_461693_object100:head# 2024-04-25T09:24:26.362 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.412 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6590eadb:::benchmark_data_smithi102_461693_object159:head# 2024-04-25T09:24:26.412 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.462 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:665d634c:::benchmark_data_smithi102_461693_object90:head# 2024-04-25T09:24:26.463 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.504 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6748bc13:::benchmark_data_smithi102_461693_object36:head# 2024-04-25T09:24:26.504 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.546 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6785c648:::benchmark_data_smithi102_461693_object1:head# 2024-04-25T09:24:26.546 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.588 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6823a015:::benchmark_data_smithi102_461693_object254:head# 2024-04-25T09:24:26.588 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.630 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68498f01:::benchmark_data_smithi102_461693_object280:head# 2024-04-25T09:24:26.630 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.671 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68811066:::benchmark_data_smithi102_461693_object163:head# 2024-04-25T09:24:26.671 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.722 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68b09bf8:::benchmark_data_smithi102_461693_object224:head# 2024-04-25T09:24:26.722 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.763 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68c982f8:::benchmark_data_smithi102_461693_object97:head# 2024-04-25T09:24:26.764 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.837 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:69b24405:::benchmark_data_smithi102_461693_object294:head# 2024-04-25T09:24:26.837 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.872 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:69e50a6f:::benchmark_data_smithi102_461693_object143:head# 2024-04-25T09:24:26.872 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.914 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6a328a99:::benchmark_data_smithi102_461693_object120:head# 2024-04-25T09:24:26.914 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.956 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6aea4f95:::benchmark_data_smithi102_461693_object71:head# 2024-04-25T09:24:26.956 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:26.997 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6b81e8fb:::benchmark_data_smithi102_461693_object140:head# 2024-04-25T09:24:26.998 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.039 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6ba5d254:::benchmark_data_smithi102_461693_object68:head# 2024-04-25T09:24:27.039 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.081 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6bf4899e:::benchmark_data_smithi102_461693_object116:head# 2024-04-25T09:24:27.081 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.123 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6d1b960b:::benchmark_data_smithi102_461693_object35:head# 2024-04-25T09:24:27.123 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.165 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6df62e7a:::benchmark_data_smithi102_461693_object56:head# 2024-04-25T09:24:27.165 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.206 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6fd57c98:::benchmark_data_smithi102_461693_object162:head# 2024-04-25T09:24:27.206 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.248 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:70d697fa:::benchmark_data_smithi102_461693_object297:head# 2024-04-25T09:24:27.248 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.290 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:72074391:::benchmark_data_smithi102_461693_object102:head# 2024-04-25T09:24:27.290 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.482 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:74228ab3:::benchmark_data_smithi102_461693_object313:head# 2024-04-25T09:24:27.482 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.524 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:745a30ad:::benchmark_data_smithi102_461693_object75:head# 2024-04-25T09:24:27.524 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.565 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:762df197:::benchmark_data_smithi102_461693_object241:head# 2024-04-25T09:24:27.566 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.649 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7744fca5:::benchmark_data_smithi102_461693_object318:head# 2024-04-25T09:24:27.650 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.708 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:77ffafc4:::benchmark_data_smithi102_461693_object24:head# 2024-04-25T09:24:27.708 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.749 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:78013faf:::benchmark_data_smithi102_461693_object175:head# 2024-04-25T09:24:27.749 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.791 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7902734b:::benchmark_data_smithi102_461693_object135:head# 2024-04-25T09:24:27.791 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.833 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7958dd60:::benchmark_data_smithi102_461693_object190:head# 2024-04-25T09:24:27.833 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.875 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7a196166:::benchmark_data_smithi102_461693_object309:head# 2024-04-25T09:24:27.875 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.917 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7a932c37:::benchmark_data_smithi102_461693_object179:head# 2024-04-25T09:24:27.917 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:27.959 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7af1be98:::benchmark_data_smithi102_461693_object298:head# 2024-04-25T09:24:27.959 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:28.017 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7b48691d:::benchmark_data_smithi102_461693_object79:head# 2024-04-25T09:24:28.017 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:28.059 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7cc3111f:::benchmark_data_smithi102_461693_object44:head# 2024-04-25T09:24:28.059 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:28.113 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7cf5ad47:::benchmark_data_smithi102_461693_object121:head# 2024-04-25T09:24:28.113 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:28.205 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7d1cce5a:::benchmark_data_smithi102_461693_object206:head# 2024-04-25T09:24:28.205 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:28.285 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7dca102f:::benchmark_data_smithi102_461693_object202:head# 2024-04-25T09:24:28.285 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:28.327 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7df4b946:::benchmark_data_smithi102_461693_object245:head# 2024-04-25T09:24:28.327 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:28.368 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7e06601e:::benchmark_data_smithi102_461693_object252:head# 2024-04-25T09:24:28.369 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:28.410 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7f86f02e:::benchmark_data_smithi102_461693_object138:head# 2024-04-25T09:24:28.411 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:28.452 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7fb155ff:::benchmark_data_smithi102_461693_object188:head# 2024-04-25T09:24:28.452 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:24:28.494 INFO:tasks.workunit.client.0.smithi102.stdout:write_pg epoch 10 info 1.0( v 10'176 (0'0,10'176] local-lis/les=7/8 n=176 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:24:28.544 INFO:tasks.workunit.client.0.smithi102.stdout:Import successful 2024-04-25T09:24:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:115: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:24:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:24:29.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:24:29.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:24:29.085 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:24:29.085 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:24:29.085 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:29.085 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:24:29.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:24:29.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:24:29.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:24:29.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:24:29.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:24:29.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:24:29.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:24:29.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:24:29.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:24:29.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:24:29.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:24:29.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:24:29.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:24:29.087 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:24:29.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:24:29.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:24:29.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:24:29.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:24:29.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:24:29.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:24:29.122 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:24:29.135+0000 7ff4d37377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:29.125 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:24:29.139+0000 7ff4d37377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:29.127 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:24:29.143+0000 7ff4d37377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:24:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:24:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:24:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:24:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:24:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:29.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:24:29.500 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:24:29.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:29.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:29.684 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:24:29.699+0000 7ff4d37377c0 -1 Falling back to public interface 2024-04-25T09:24:29.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:30.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:30.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:30.935 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:24:30.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:24:30.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:30.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:31.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:32.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:32.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:32.344 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:24:32.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:24:32.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:32.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:32.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:33.051 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:24:33.067+0000 7ff4d37377c0 -1 osd.0 21 log_to_monitors true 2024-04-25T09:24:33.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:33.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:33.763 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:24:33.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:24:33.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:33.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:24:34.174 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 24 up_thru 24 down_at 22 last_clean_interval [13,21) [v2:127.0.0.1:6802/2930603955,v1:127.0.0.1:6803/2930603955] [v2:127.0.0.1:6804/2930603955,v1:127.0.0.1:6805/2930603955] exists,up 150f6751-a5c0-42ad-b859-7e7a4382302c 2024-04-25T09:24:34.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:24:34.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:24:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:24:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:117: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:24:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:24:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:24:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:24:34.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:24:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:24:34.176 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:24:34.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:24:34.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:24:34.176 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:24:34.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:24:34.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:24:34.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:24:34.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:24:34.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:24:34.277 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:24:34.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:24:34.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:24:34.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:34.697 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:24:34.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215107 2024-04-25T09:24:34.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215107 2024-04-25T09:24:34.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215107' 2024-04-25T09:24:34.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:34.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215107 2024-04-25T09:24:34.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:34.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:24:34.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215107 2024-04-25T09:24:34.876 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:34.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215107 2024-04-25T09:24:34.877 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 103079215107 2024-04-25T09:24:34.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215107' 2024-04-25T09:24:34.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:24:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 103079215107 2024-04-25T09:24:35.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:24:36.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:24:36.276 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:24:36.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215107 -lt 103079215107 2024-04-25T09:24:36.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:24:36.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:36.711 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:37.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:24:37.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:24:37.273 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:24:37.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:24:37.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:24:37.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:24:37.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:24:37.274 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:24:37.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:24:37.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:24:37.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:37.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:38.179 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:24:38.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:24:38.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:24:38.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: seq 1 50 2024-04-25T09:24:38.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:38.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:38.854 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:38.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:39.824 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:39.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:39.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:40.721 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:40.745 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:41.406 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:41.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:41.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:42.209 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:42.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:42.954 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:42.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:42.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:43.824 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:43.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:44.577 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:44.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:44.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:45.414 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:45.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:46.234 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:46.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:46.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:46.987 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:47.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:47.791 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:47.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:47.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:48.485 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:48.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:49.255 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:49.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:49.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:50.042 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:50.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:50.677 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:50.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:50.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:51.498 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:51.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:52.309 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:52.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:52.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:53.071 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:53.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:53.882 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:53.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:53.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:54.702 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:54.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:55.523 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:55.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:55.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:56.318 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:56.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:57.088 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:57.110 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:57.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:57.949 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:57.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:24:58.652 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:24:58.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:24:58.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:24:59.430 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:24:59.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:00.284 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:00.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:00.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:01.096 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:01.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:01.891 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:01.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:01.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:02.737 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:02.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:03.583 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:03.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:03.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:04.269 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:04.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:05.064 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:05.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:05.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:05.901 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:05.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:06.722 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:06.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:06.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:07.466 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:07.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:08.261 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:08.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:08.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:09.108 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:09.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:10.003 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:10.023 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:10.024 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:10.774 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:10.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:11.627 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:11.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:11.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:12.373 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:12.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:13.251 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:13.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:13.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:13.988 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:14.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:14.800 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:14.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:15.620 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:15.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:16.424 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:16.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:16.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:17.261 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:17.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:18.039 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:18.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:18.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:18.876 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:18.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:19.697 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:19.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:19.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:20.509 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:20.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:21.321 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:21.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:21.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:22.208 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:22.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:23.053 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:23.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:23.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:23.873 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:23.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:24.710 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:24.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:24.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:25.707 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:25.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:26.410 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:26.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:26.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:27.213 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:27.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:28.017 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:28.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:28.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:28.837 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:28.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:29.573 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:29.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:29.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:30.410 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:30.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:31.197 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:31.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:31.218 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:31.950 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:31.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:32.762 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:32.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:32.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:33.574 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:33.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:34.360 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:34.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:34.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:35.046 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:35.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:35.816 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:35.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:35.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:36.511 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:36.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:37.339 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:37.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:37.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:38.193 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:38.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:39.021 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:39.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:39.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:39.850 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:39.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:40.486 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:40.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:40.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:41.172 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:41.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:41.841 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:41.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:41.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:42.645 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:42.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:43.666 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:43.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:43.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:44.510 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:44.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:45.473 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:45.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:45.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:46.185 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:46.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:47.021 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:47.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:47.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:47.825 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:47.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:48.503 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:48.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:48.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:49.407 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:49.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:50.135 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:50.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:50.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:50.938 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:50.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:51.775 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:51.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:51.798 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:52.511 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:52.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:53.432 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:53.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:53.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:54.268 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:54.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:55.189 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:55.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:55.212 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:55.975 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:55.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:56.745 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:56.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:25:56.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:25:57.557 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:25:57.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:25:58.235 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:25:58.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:127: TEST_import_after_merge_and_gap: rados -p foo bench 1 write -b 1024 --no-cleanup 2024-04-25T09:25:58.306 INFO:tasks.workunit.client.0.smithi102.stdout:hints = 1 2024-04-25T09:25:58.307 INFO:tasks.workunit.client.0.smithi102.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 1 seconds or 0 objects 2024-04-25T09:25:58.307 INFO:tasks.workunit.client.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_467209 2024-04-25T09:25:58.307 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:25:58.307 INFO:tasks.workunit.client.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:25:59.306 INFO:tasks.workunit.client.0.smithi102.stdout: 1 16 160 144 0.14061 0.140625 0.101581 0.108023 2024-04-25T09:26:00.307 INFO:tasks.workunit.client.0.smithi102.stdout:Total time run: 1.06798 2024-04-25T09:26:00.307 INFO:tasks.workunit.client.0.smithi102.stdout:Total writes made: 160 2024-04-25T09:26:00.308 INFO:tasks.workunit.client.0.smithi102.stdout:Write size: 1024 2024-04-25T09:26:00.308 INFO:tasks.workunit.client.0.smithi102.stdout:Object size: 1024 2024-04-25T09:26:00.308 INFO:tasks.workunit.client.0.smithi102.stdout:Bandwidth (MB/sec): 0.146305 2024-04-25T09:26:00.308 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Bandwidth: 0 2024-04-25T09:26:00.308 INFO:tasks.workunit.client.0.smithi102.stdout:Max bandwidth (MB/sec): 0.140625 2024-04-25T09:26:00.308 INFO:tasks.workunit.client.0.smithi102.stdout:Min bandwidth (MB/sec): 0.140625 2024-04-25T09:26:00.308 INFO:tasks.workunit.client.0.smithi102.stdout:Average IOPS: 149 2024-04-25T09:26:00.309 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev IOPS: 0 2024-04-25T09:26:00.309 INFO:tasks.workunit.client.0.smithi102.stdout:Max IOPS: 144 2024-04-25T09:26:00.309 INFO:tasks.workunit.client.0.smithi102.stdout:Min IOPS: 144 2024-04-25T09:26:00.309 INFO:tasks.workunit.client.0.smithi102.stdout:Average Latency(s): 0.106351 2024-04-25T09:26:00.309 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Latency(s): 0.0283828 2024-04-25T09:26:00.309 INFO:tasks.workunit.client.0.smithi102.stdout:Max latency(s): 0.173747 2024-04-25T09:26:00.309 INFO:tasks.workunit.client.0.smithi102.stdout:Min latency(s): 0.0749824 2024-04-25T09:26:00.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:128: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:26:00.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:26:00.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:26:00.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:26:00.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:26:00.355 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:26:00.355 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:26:00.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:26:00.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:26:00.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:26:00.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:26:00.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:26:00.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:26:00.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:26:00.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:26:00.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:26:00.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:26:00.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:26:00.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:26:00.915 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:26:01.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215126 2024-04-25T09:26:01.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215126 2024-04-25T09:26:01.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-103079215126' 2024-04-25T09:26:01.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:26:01.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-103079215126 2024-04-25T09:26:01.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:26:01.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:26:01.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-103079215126 2024-04-25T09:26:01.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:26:01.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215126 2024-04-25T09:26:01.088 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 103079215126 2024-04-25T09:26:01.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 103079215126' 2024-04-25T09:26:01.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:26:01.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215124 -lt 103079215126 2024-04-25T09:26:01.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:26:02.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:26:02.493 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:26:02.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215126 -lt 103079215126 2024-04-25T09:26:02.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:26:02.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:26:02.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:26:03.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:26:03.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:26:03.487 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:26:03.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:26:03.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:26:03.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:26:03.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:26:03.488 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:26:03.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:26:03.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:26:03.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:26:03.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:26:04.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:26:04.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:26:04.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:26:04.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:129: TEST_import_after_merge_and_gap: ceph tell osd.0 send_beacon 2024-04-25T09:26:04.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:130: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:26:09.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:131: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:26:10.413 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is set 2024-04-25T09:26:10.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:132: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:26:11.225 INFO:tasks.workunit.client.0.smithi102.stderr:nodown is unset 2024-04-25T09:26:11.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:133: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:26:16.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:135: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:26:16.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:26:16.246 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:26:16.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:26:16.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:26:16.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:26:16.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:26:16.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:138: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:26:17.717 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:26:17.717 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.1 2024-04-25T09:26:17.718 INFO:tasks.workunit.client.0.smithi102.stderr:PG export's map 10 is older than OSD's oldest_map 61 2024-04-25T09:26:17.718 INFO:tasks.workunit.client.0.smithi102.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:26:18.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:140: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export-remove --pgid 1.0 --force --file td/pg-split-merge/1.0.later 2024-04-25T09:26:19.256 INFO:tasks.workunit.client.0.smithi102.stderr:Exporting 1.0 info 1.0( v 225'337 (10'176,225'337] local-lis/les=24/25 n=481 ec=7/7 lis/c=24/24 les/c/f=25/25/0 sis=24) 2024-04-25T09:26:19.273 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:000b3b92:::benchmark_data_smithi102_461693_object262:head# 2024-04-25T09:26:19.274 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:00374c2e:::benchmark_data_smithi102_461693_object216:head# 2024-04-25T09:26:19.274 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:01237c3c:::benchmark_data_smithi102_467209_object147:head# 2024-04-25T09:26:19.275 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0233f50b:::benchmark_data_smithi102_467209_object54:head# 2024-04-25T09:26:19.276 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0301c80c:::benchmark_data_smithi102_461693_object11:head# 2024-04-25T09:26:19.276 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:036651e6:::benchmark_data_smithi102_461693_object111:head# 2024-04-25T09:26:19.277 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0402916e:::benchmark_data_smithi102_461693_object183:head# 2024-04-25T09:26:19.277 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:046b3940:::benchmark_data_smithi102_461693_object104:head# 2024-04-25T09:26:19.278 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:046d0875:::benchmark_data_smithi102_467209_object139:head# 2024-04-25T09:26:19.279 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0540312f:::benchmark_data_smithi102_461693_object69:head# 2024-04-25T09:26:19.279 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:054ee2e1:::benchmark_data_smithi102_461693_object300:head# 2024-04-25T09:26:19.280 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:062f7c5e:::benchmark_data_smithi102_467209_object32:head# 2024-04-25T09:26:19.280 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:069b73e0:::benchmark_data_smithi102_461693_object195:head# 2024-04-25T09:26:19.281 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:06a51e53:::benchmark_data_smithi102_467209_object137:head# 2024-04-25T09:26:19.282 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:06bf8ef0:::benchmark_data_smithi102_467209_object156:head# 2024-04-25T09:26:19.282 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:07526ebf:::benchmark_data_smithi102_461693_object248:head# 2024-04-25T09:26:19.283 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0825096c:::benchmark_data_smithi102_467209_object15:head# 2024-04-25T09:26:19.284 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:084f14dd:::benchmark_data_smithi102_467209_object77:head# 2024-04-25T09:26:19.284 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:08aeff1a:::benchmark_data_smithi102_461693_object43:head# 2024-04-25T09:26:19.285 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:09569df1:::benchmark_data_smithi102_461693_object305:head# 2024-04-25T09:26:19.286 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:099020c4:::benchmark_data_smithi102_461693_object152:head# 2024-04-25T09:26:19.286 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:09a7c99f:::benchmark_data_smithi102_467209_object1:head# 2024-04-25T09:26:19.287 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0a47cb3d:::benchmark_data_smithi102_461693_object266:head# 2024-04-25T09:26:19.287 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0a7de0f7:::benchmark_data_smithi102_461693_object208:head# 2024-04-25T09:26:19.288 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0b43c519:::benchmark_data_smithi102_467209_object42:head# 2024-04-25T09:26:19.288 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0b6a70be:::benchmark_data_smithi102_461693_object85:head# 2024-04-25T09:26:19.289 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0c02d064:::benchmark_data_smithi102_467209_object112:head# 2024-04-25T09:26:19.290 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0cc78a03:::benchmark_data_smithi102_461693_object21:head# 2024-04-25T09:26:19.298 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0cd9cd5d:::benchmark_data_smithi102_461693_object160:head# 2024-04-25T09:26:19.302 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0cfde1d2:::benchmark_data_smithi102_467209_object105:head# 2024-04-25T09:26:19.303 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0deb35d3:::benchmark_data_smithi102_461693_object217:head# 2024-04-25T09:26:19.304 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0e8b2dd0:::benchmark_data_smithi102_461693_object148:head# 2024-04-25T09:26:19.305 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0f24b073:::benchmark_data_smithi102_461693_object317:head# 2024-04-25T09:26:19.305 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0ff8e14a:::benchmark_data_smithi102_467209_object3:head# 2024-04-25T09:26:19.305 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:10af48fb:::benchmark_data_smithi102_461693_object240:head# 2024-04-25T09:26:19.306 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:11692bdb:::benchmark_data_smithi102_461693_object27:head# 2024-04-25T09:26:19.307 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:128e9745:::benchmark_data_smithi102_461693_object314:head# 2024-04-25T09:26:19.307 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:12d5a862:::benchmark_data_smithi102_467209_object107:head# 2024-04-25T09:26:19.308 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:132fbbae:::benchmark_data_smithi102_461693_object92:head# 2024-04-25T09:26:19.309 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:14a6a0cd:::benchmark_data_smithi102_461693_object88:head# 2024-04-25T09:26:19.310 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:14f57f39:::benchmark_data_smithi102_467209_object58:head# 2024-04-25T09:26:19.311 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1512b06b:::benchmark_data_smithi102_461693_object272:head# 2024-04-25T09:26:19.311 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1629e659:::benchmark_data_smithi102_461693_object256:head# 2024-04-25T09:26:19.312 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:169aee60:::benchmark_data_smithi102_467209_object88:head# 2024-04-25T09:26:19.313 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:16f7bd8c:::benchmark_data_smithi102_461693_object64:head# 2024-04-25T09:26:19.313 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:17105e23:::benchmark_data_smithi102_467209_object136:head# 2024-04-25T09:26:19.314 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1786312c:::benchmark_data_smithi102_461693_object37:head# 2024-04-25T09:26:19.314 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:18b9710d:::benchmark_data_smithi102_467209_object131:head# 2024-04-25T09:26:19.315 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:19fa802a:::benchmark_data_smithi102_461693_object177:head# 2024-04-25T09:26:19.316 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1b20ea55:::benchmark_data_smithi102_467209_object2:head# 2024-04-25T09:26:19.316 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1b6923a0:::benchmark_data_smithi102_467209_object7:head# 2024-04-25T09:26:19.317 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1ba38020:::benchmark_data_smithi102_467209_object55:head# 2024-04-25T09:26:19.317 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1c114184:::benchmark_data_smithi102_467209_object8:head# 2024-04-25T09:26:19.318 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1c85f9b4:::benchmark_data_smithi102_467209_object25:head# 2024-04-25T09:26:19.318 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1d2fbc72:::benchmark_data_smithi102_461693_object253:head# 2024-04-25T09:26:19.319 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1dbe156c:::benchmark_data_smithi102_461693_object301:head# 2024-04-25T09:26:19.319 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1e43a575:::benchmark_data_smithi102_461693_object109:head# 2024-04-25T09:26:19.320 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1e98b968:::benchmark_data_smithi102_461693_object8:head# 2024-04-25T09:26:19.321 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1e9976c2:::benchmark_data_smithi102_461693_object31:head# 2024-04-25T09:26:19.321 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:202535ea:::benchmark_data_smithi102_467209_object121:head# 2024-04-25T09:26:19.322 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:20ac13d7:::benchmark_data_smithi102_461693_object319:head# 2024-04-25T09:26:19.323 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:217c431d:::benchmark_data_smithi102_467209_object84:head# 2024-04-25T09:26:19.324 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2279d77a:::benchmark_data_smithi102_461693_object157:head# 2024-04-25T09:26:19.324 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:22b40e23:::benchmark_data_smithi102_461693_object233:head# 2024-04-25T09:26:19.325 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:23111692:::benchmark_data_smithi102_461693_object14:head# 2024-04-25T09:26:19.325 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2331325a:::benchmark_data_smithi102_467209_object70:head# 2024-04-25T09:26:19.326 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:23426916:::benchmark_data_smithi102_461693_object164:head# 2024-04-25T09:26:19.326 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:238a3ae3:::benchmark_data_smithi102_461693_object211:head# 2024-04-25T09:26:19.327 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:239e27b3:::benchmark_data_smithi102_461693_object219:head# 2024-04-25T09:26:19.328 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:23b4b493:::benchmark_data_smithi102_461693_object192:head# 2024-04-25T09:26:19.328 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:24e58257:::benchmark_data_smithi102_461693_object141:head# 2024-04-25T09:26:19.329 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:24e6a109:::benchmark_data_smithi102_461693_object106:head# 2024-04-25T09:26:19.330 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:25506ed7:::benchmark_data_smithi102_467209_object100:head# 2024-04-25T09:26:19.331 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:25997359:::benchmark_data_smithi102_461693_object264:head# 2024-04-25T09:26:19.332 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:25b16926:::benchmark_data_smithi102_461693_object29:head# 2024-04-25T09:26:19.332 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:25e5150c:::benchmark_data_smithi102_467209_object126:head# 2024-04-25T09:26:19.333 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:26824afa:::benchmark_data_smithi102_461693_object94:head# 2024-04-25T09:26:19.333 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:26a55470:::benchmark_data_smithi102_461693_object274:head# 2024-04-25T09:26:19.334 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:272eb2c9:::benchmark_data_smithi102_461693_object149:head# 2024-04-25T09:26:19.335 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:273627fb:::benchmark_data_smithi102_461693_object125:head# 2024-04-25T09:26:19.335 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:273b3e12:::benchmark_data_smithi102_461693_object161:head# 2024-04-25T09:26:19.336 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:275d130a:::benchmark_data_smithi102_461693_object20:head# 2024-04-25T09:26:19.336 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:28694220:::benchmark_data_smithi102_461693_object136:head# 2024-04-25T09:26:19.337 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2a0bbe58:::benchmark_data_smithi102_461693_object199:head# 2024-04-25T09:26:19.337 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2a74d816:::benchmark_data_smithi102_467209_object99:head# 2024-04-25T09:26:19.338 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2a86184a:::benchmark_data_smithi102_467209_object39:head# 2024-04-25T09:26:19.339 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2afc2f48:::benchmark_data_smithi102_461693_object209:head# 2024-04-25T09:26:19.339 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2bbb9370:::benchmark_data_smithi102_461693_object287:head# 2024-04-25T09:26:19.340 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2bd520cd:::benchmark_data_smithi102_461693_object19:head# 2024-04-25T09:26:19.341 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2be202af:::benchmark_data_smithi102_461693_object234:head# 2024-04-25T09:26:19.342 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2c0b4651:::benchmark_data_smithi102_461693_object74:head# 2024-04-25T09:26:19.342 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2c9a727c:::benchmark_data_smithi102_461693_object165:head# 2024-04-25T09:26:19.343 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2cf2f038:::benchmark_data_smithi102_467209_object118:head# 2024-04-25T09:26:19.344 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2d3d0890:::benchmark_data_smithi102_461693_object50:head# 2024-04-25T09:26:19.345 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2d3fef0a:::benchmark_data_smithi102_461693_object173:head# 2024-04-25T09:26:19.345 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2d4d8992:::benchmark_data_smithi102_461693_object53:head# 2024-04-25T09:26:19.346 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2d554bd3:::benchmark_data_smithi102_467209_object65:head# 2024-04-25T09:26:19.346 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2daabe1c:::benchmark_data_smithi102_467209_object44:head# 2024-04-25T09:26:19.347 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2ee095c4:::benchmark_data_smithi102_461693_object58:head# 2024-04-25T09:26:19.348 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2fd07b30:::benchmark_data_smithi102_461693_object194:head# 2024-04-25T09:26:19.348 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2fee0198:::benchmark_data_smithi102_461693_object2:head# 2024-04-25T09:26:19.348 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:31f0e698:::benchmark_data_smithi102_467209_object6:head# 2024-04-25T09:26:19.349 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:34891980:::benchmark_data_smithi102_467209_object0:head# 2024-04-25T09:26:19.349 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:34e9f16c:::benchmark_data_smithi102_461693_object260:head# 2024-04-25T09:26:19.350 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:35004fae:::benchmark_data_smithi102_467209_object68:head# 2024-04-25T09:26:19.350 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:355453a0:::benchmark_data_smithi102_461693_object118:head# 2024-04-25T09:26:19.351 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:355b9f1f:::benchmark_data_smithi102_467209_object110:head# 2024-04-25T09:26:19.351 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:363cd543:::benchmark_data_smithi102_461693_object34:head# 2024-04-25T09:26:19.352 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:36551f3c:::benchmark_data_smithi102_461693_object51:head# 2024-04-25T09:26:19.353 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3693711a:::benchmark_data_smithi102_461693_object306:head# 2024-04-25T09:26:19.353 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:37225bcc:::benchmark_data_smithi102_461693_object117:head# 2024-04-25T09:26:19.354 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:37c0cca3:::benchmark_data_smithi102_461693_object282:head# 2024-04-25T09:26:19.354 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:389b2f68:::benchmark_data_smithi102_461693_object54:head# 2024-04-25T09:26:19.355 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:38c55045:::benchmark_data_smithi102_467209_object143:head# 2024-04-25T09:26:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:38f5ec38:::benchmark_data_smithi102_461693_object12:head# 2024-04-25T09:26:19.356 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:396efdd2:::benchmark_data_smithi102_467209_object63:head# 2024-04-25T09:26:19.357 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:39a2e424:::benchmark_data_smithi102_461693_object16:head# 2024-04-25T09:26:19.358 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3b46a605:::benchmark_data_smithi102_467209_object57:head# 2024-04-25T09:26:19.358 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3b72c6d5:::benchmark_data_smithi102_461693_object28:head# 2024-04-25T09:26:19.359 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3b87185e:::benchmark_data_smithi102_467209_object45:head# 2024-04-25T09:26:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3bca4278:::benchmark_data_smithi102_461693_object6:head# 2024-04-25T09:26:19.360 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3c246a75:::benchmark_data_smithi102_467209_object83:head# 2024-04-25T09:26:19.361 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3c78f5d8:::benchmark_data_smithi102_461693_object124:head# 2024-04-25T09:26:19.361 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3ccb0f2b:::benchmark_data_smithi102_461693_object144:head# 2024-04-25T09:26:19.362 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3d133806:::benchmark_data_smithi102_461693_object243:head# 2024-04-25T09:26:19.362 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3d2ad51e:::benchmark_data_smithi102_461693_object38:head# 2024-04-25T09:26:19.363 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3d611f0a:::benchmark_data_smithi102_461693_object32:head# 2024-04-25T09:26:19.364 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3f6d6ad9:::benchmark_data_smithi102_461693_object204:head# 2024-04-25T09:26:19.364 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3f7d2427:::benchmark_data_smithi102_461693_object205:head# 2024-04-25T09:26:19.365 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:40581110:::benchmark_data_smithi102_461693_object147:head# 2024-04-25T09:26:19.365 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4066b8f9:::benchmark_data_smithi102_461693_object103:head# 2024-04-25T09:26:19.365 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:407cca9e:::benchmark_data_smithi102_461693_object228:head# 2024-04-25T09:26:19.366 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:411219b0:::benchmark_data_smithi102_467209_object154:head# 2024-04-25T09:26:19.366 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:411befe3:::benchmark_data_smithi102_461693_object25:head# 2024-04-25T09:26:19.367 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:412d23d7:::benchmark_data_smithi102_461693_object174:head# 2024-04-25T09:26:19.368 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:426f5982:::benchmark_data_smithi102_461693_object7:head# 2024-04-25T09:26:19.368 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:44133081:::benchmark_data_smithi102_467209_object69:head# 2024-04-25T09:26:19.369 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:443e3636:::benchmark_data_smithi102_461693_object201:head# 2024-04-25T09:26:19.370 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:44a791f4:::benchmark_data_smithi102_467209_object146:head# 2024-04-25T09:26:19.370 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:45dbb99e:::benchmark_data_smithi102_461693_object139:head# 2024-04-25T09:26:19.371 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:462c230e:::benchmark_data_smithi102_461693_object189:head# 2024-04-25T09:26:19.372 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:46e420c2:::benchmark_data_smithi102_461693_object315:head# 2024-04-25T09:26:19.372 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:46ed67bb:::benchmark_data_smithi102_461693_object80:head# 2024-04-25T09:26:19.373 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:47d29e85:::benchmark_data_smithi102_461693_object89:head# 2024-04-25T09:26:19.373 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:485eaaa8:::benchmark_data_smithi102_461693_object66:head# 2024-04-25T09:26:19.374 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:49105468:::benchmark_data_smithi102_461693_object76:head# 2024-04-25T09:26:19.374 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:49d9b3e4:::benchmark_data_smithi102_461693_object268:head# 2024-04-25T09:26:19.375 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4a341421:::benchmark_data_smithi102_461693_object288:head# 2024-04-25T09:26:19.375 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4a657f62:::benchmark_data_smithi102_467209_object132:head# 2024-04-25T09:26:19.376 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4b1ffa35:::benchmark_data_smithi102_461693_object286:head# 2024-04-25T09:26:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4cab95a8:::benchmark_data_smithi102_467209_object51:head# 2024-04-25T09:26:19.377 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4cfe58df:::benchmark_data_smithi102_467209_object122:head# 2024-04-25T09:26:19.378 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4d5b2900:::benchmark_data_smithi102_461693_object23:head# 2024-04-25T09:26:19.378 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4d6d7ce3:::benchmark_data_smithi102_461693_object122:head# 2024-04-25T09:26:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4e5a5bae:::benchmark_data_smithi102_461693_object214:head# 2024-04-25T09:26:19.379 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4ee4fe84:::benchmark_data_smithi102_461693_object63:head# 2024-04-25T09:26:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4f022233:::benchmark_data_smithi102_467209_object81:head# 2024-04-25T09:26:19.380 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4f4aa74e:::benchmark_data_smithi102_461693_object146:head# 2024-04-25T09:26:19.381 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4f7e5631:::benchmark_data_smithi102_461693_object251:head# 2024-04-25T09:26:19.382 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4f9e0719:::benchmark_data_smithi102_461693_object222:head# 2024-04-25T09:26:19.382 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:507c4ede:::benchmark_data_smithi102_467209_object11:head# 2024-04-25T09:26:19.383 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:51ef0d4b:::benchmark_data_smithi102_467209_object75:head# 2024-04-25T09:26:19.383 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:52c86389:::benchmark_data_smithi102_461693_object223:head# 2024-04-25T09:26:19.384 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5308d886:::benchmark_data_smithi102_461693_object45:head# 2024-04-25T09:26:19.384 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5403a8e1:::benchmark_data_smithi102_461693_object276:head# 2024-04-25T09:26:19.385 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:54221da9:::benchmark_data_smithi102_467209_object59:head# 2024-04-25T09:26:19.385 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:547cc0c2:::benchmark_data_smithi102_461693_object169:head# 2024-04-25T09:26:19.386 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:54f5df8b:::benchmark_data_smithi102_461693_object98:head# 2024-04-25T09:26:19.386 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5503c0c7:::benchmark_data_smithi102_461693_object133:head# 2024-04-25T09:26:19.387 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:557ea668:::benchmark_data_smithi102_461693_object65:head# 2024-04-25T09:26:19.387 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:56c5df50:::benchmark_data_smithi102_467209_object61:head# 2024-04-25T09:26:19.388 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:57f2a07b:::benchmark_data_smithi102_461693_object293:head# 2024-04-25T09:26:19.388 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:580d47d3:::benchmark_data_smithi102_461693_object10:head# 2024-04-25T09:26:19.389 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:58a91fd0:::benchmark_data_smithi102_461693_object292:head# 2024-04-25T09:26:19.389 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5a9a4702:::benchmark_data_smithi102_461693_object30:head# 2024-04-25T09:26:19.390 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5b1e4f29:::benchmark_data_smithi102_467209_object76:head# 2024-04-25T09:26:19.390 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5b7a7287:::benchmark_data_smithi102_461693_object178:head# 2024-04-25T09:26:19.391 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5c075ec8:::benchmark_data_smithi102_461693_object91:head# 2024-04-25T09:26:19.391 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5ce181a5:::benchmark_data_smithi102_467209_object134:head# 2024-04-25T09:26:19.392 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5d2a150c:::benchmark_data_smithi102_467209_object125:head# 2024-04-25T09:26:19.392 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5d5f48ea:::benchmark_data_smithi102_461693_object302:head# 2024-04-25T09:26:19.393 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5d8f2267:::benchmark_data_smithi102_461693_object247:head# 2024-04-25T09:26:19.393 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5e9e18aa:::benchmark_data_smithi102_467209_object123:head# 2024-04-25T09:26:19.394 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5ec1726b:::benchmark_data_smithi102_461693_object126:head# 2024-04-25T09:26:19.395 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5ecc9427:::benchmark_data_smithi102_467209_object127:head# 2024-04-25T09:26:19.395 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5fc8778c:::benchmark_data_smithi102_467209_object155:head# 2024-04-25T09:26:19.396 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5fe7afcc:::benchmark_data_smithi102_461693_object238:head# 2024-04-25T09:26:19.396 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6009192c:::benchmark_data_smithi102_467209_object157:head# 2024-04-25T09:26:19.397 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:60ff7266:::benchmark_data_smithi102_461693_object42:head# 2024-04-25T09:26:19.397 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:61051e5f:::benchmark_data_smithi102_461693_object128:head# 2024-04-25T09:26:19.398 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:638903a9:::benchmark_data_smithi102_461693_object271:head# 2024-04-25T09:26:19.398 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:64392431:::benchmark_data_smithi102_461693_object316:head# 2024-04-25T09:26:19.399 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6452bd93:::benchmark_data_smithi102_461693_object78:head# 2024-04-25T09:26:19.400 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6469f64c:::benchmark_data_smithi102_461693_object246:head# 2024-04-25T09:26:19.400 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:64ce6296:::benchmark_data_smithi102_461693_object100:head# 2024-04-25T09:26:19.401 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6590eadb:::benchmark_data_smithi102_461693_object159:head# 2024-04-25T09:26:19.401 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:665d634c:::benchmark_data_smithi102_461693_object90:head# 2024-04-25T09:26:19.402 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6748bc13:::benchmark_data_smithi102_461693_object36:head# 2024-04-25T09:26:19.402 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6785c648:::benchmark_data_smithi102_461693_object1:head# 2024-04-25T09:26:19.402 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6823a015:::benchmark_data_smithi102_461693_object254:head# 2024-04-25T09:26:19.402 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:68498f01:::benchmark_data_smithi102_461693_object280:head# 2024-04-25T09:26:19.402 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:68811066:::benchmark_data_smithi102_461693_object163:head# 2024-04-25T09:26:19.403 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:68b09bf8:::benchmark_data_smithi102_461693_object224:head# 2024-04-25T09:26:19.403 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:68c982f8:::benchmark_data_smithi102_461693_object97:head# 2024-04-25T09:26:19.403 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:69b24405:::benchmark_data_smithi102_461693_object294:head# 2024-04-25T09:26:19.404 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:69e50a6f:::benchmark_data_smithi102_461693_object143:head# 2024-04-25T09:26:19.404 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6a328a99:::benchmark_data_smithi102_461693_object120:head# 2024-04-25T09:26:19.405 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6aea4f95:::benchmark_data_smithi102_461693_object71:head# 2024-04-25T09:26:19.405 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6b81e8fb:::benchmark_data_smithi102_461693_object140:head# 2024-04-25T09:26:19.405 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6ba5d254:::benchmark_data_smithi102_461693_object68:head# 2024-04-25T09:26:19.406 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6bb50989:::benchmark_data_smithi102_467209_object9:head# 2024-04-25T09:26:19.406 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6bf4899e:::benchmark_data_smithi102_461693_object116:head# 2024-04-25T09:26:19.407 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6c3afad8:::benchmark_data_smithi102_467209_object50:head# 2024-04-25T09:26:19.407 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6d1b960b:::benchmark_data_smithi102_461693_object35:head# 2024-04-25T09:26:19.408 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6df62e7a:::benchmark_data_smithi102_461693_object56:head# 2024-04-25T09:26:19.409 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6e487e28:::benchmark_data_smithi102_467209_object129:head# 2024-04-25T09:26:19.409 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6f5152dc:::benchmark_data_smithi102_467209_object14:head# 2024-04-25T09:26:19.410 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6fd57c98:::benchmark_data_smithi102_461693_object162:head# 2024-04-25T09:26:19.410 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:70b3821b:::benchmark_data_smithi102_467209_object60:head# 2024-04-25T09:26:19.411 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:70d697fa:::benchmark_data_smithi102_461693_object297:head# 2024-04-25T09:26:19.411 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:71abbb27:::benchmark_data_smithi102_467209_object19:head# 2024-04-25T09:26:19.412 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:72074391:::benchmark_data_smithi102_461693_object102:head# 2024-04-25T09:26:19.412 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:74228ab3:::benchmark_data_smithi102_461693_object313:head# 2024-04-25T09:26:19.413 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:745a30ad:::benchmark_data_smithi102_461693_object75:head# 2024-04-25T09:26:19.414 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:75aa73ef:::benchmark_data_smithi102_467209_object20:head# 2024-04-25T09:26:19.414 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:762df197:::benchmark_data_smithi102_461693_object241:head# 2024-04-25T09:26:19.415 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7744fca5:::benchmark_data_smithi102_461693_object318:head# 2024-04-25T09:26:19.415 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:779d6c73:::benchmark_data_smithi102_467209_object153:head# 2024-04-25T09:26:19.416 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:77ae3015:::benchmark_data_smithi102_467209_object78:head# 2024-04-25T09:26:19.416 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:77dae1b7:::benchmark_data_smithi102_467209_object30:head# 2024-04-25T09:26:19.417 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:77ffafc4:::benchmark_data_smithi102_461693_object24:head# 2024-04-25T09:26:19.417 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:78013faf:::benchmark_data_smithi102_461693_object175:head# 2024-04-25T09:26:19.418 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7902734b:::benchmark_data_smithi102_461693_object135:head# 2024-04-25T09:26:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7958dd60:::benchmark_data_smithi102_461693_object190:head# 2024-04-25T09:26:19.419 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7a196166:::benchmark_data_smithi102_461693_object309:head# 2024-04-25T09:26:19.420 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7a729e51:::benchmark_data_smithi102_467209_object130:head# 2024-04-25T09:26:19.421 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7a932c37:::benchmark_data_smithi102_461693_object179:head# 2024-04-25T09:26:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7aea4ed6:::benchmark_data_smithi102_467209_object37:head# 2024-04-25T09:26:19.422 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7af1be98:::benchmark_data_smithi102_461693_object298:head# 2024-04-25T09:26:19.423 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7b48691d:::benchmark_data_smithi102_461693_object79:head# 2024-04-25T09:26:19.423 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7c3929ca:::benchmark_data_smithi102_467209_object85:head# 2024-04-25T09:26:19.424 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7cc3111f:::benchmark_data_smithi102_461693_object44:head# 2024-04-25T09:26:19.425 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7cf5ad47:::benchmark_data_smithi102_461693_object121:head# 2024-04-25T09:26:19.425 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7d1cce5a:::benchmark_data_smithi102_461693_object206:head# 2024-04-25T09:26:19.426 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7dca102f:::benchmark_data_smithi102_461693_object202:head# 2024-04-25T09:26:19.426 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7df4b946:::benchmark_data_smithi102_461693_object245:head# 2024-04-25T09:26:19.427 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7e06601e:::benchmark_data_smithi102_461693_object252:head# 2024-04-25T09:26:19.427 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7e8819ca:::benchmark_data_smithi102_467209_object92:head# 2024-04-25T09:26:19.429 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7f86f02e:::benchmark_data_smithi102_461693_object138:head# 2024-04-25T09:26:19.429 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7fb155ff:::benchmark_data_smithi102_461693_object188:head# 2024-04-25T09:26:19.430 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:804ec195:::benchmark_data_smithi102_467209_object16:head# 2024-04-25T09:26:19.430 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:807e8dc6:::benchmark_data_smithi102_467209_object74:head# 2024-04-25T09:26:19.431 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:808c9235:::benchmark_data_smithi102_467209_object34:head# 2024-04-25T09:26:19.431 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:80decefa:::benchmark_data_smithi102_461693_object215:head# 2024-04-25T09:26:19.432 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:81279b41:::benchmark_data_smithi102_467209_object87:head# 2024-04-25T09:26:19.433 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:821259f4:::benchmark_data_smithi102_467209_object101:head# 2024-04-25T09:26:19.433 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:82443fae:::benchmark_data_smithi102_467209_object40:head# 2024-04-25T09:26:19.434 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:82504851:::benchmark_data_smithi102_461693_object59:head# 2024-04-25T09:26:19.435 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:82bc124c:::benchmark_data_smithi102_461693_object242:head# 2024-04-25T09:26:19.435 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:83944c13:::benchmark_data_smithi102_467209_object82:head# 2024-04-25T09:26:19.436 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:83a6ed54:::benchmark_data_smithi102_467209_object120:head# 2024-04-25T09:26:19.436 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:84376e46:::benchmark_data_smithi102_467209_object12:head# 2024-04-25T09:26:19.436 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:883cb801:::benchmark_data_smithi102_461693_object150:head# 2024-04-25T09:26:19.437 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:887d5142:::benchmark_data_smithi102_467209_object151:head# 2024-04-25T09:26:19.438 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:889e9aa5:::benchmark_data_smithi102_461693_object250:head# 2024-04-25T09:26:19.438 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:88bc3744:::benchmark_data_smithi102_467209_object79:head# 2024-04-25T09:26:19.439 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:890093a8:::benchmark_data_smithi102_467209_object109:head# 2024-04-25T09:26:19.439 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:890d9aaf:::benchmark_data_smithi102_461693_object96:head# 2024-04-25T09:26:19.440 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8a093910:::benchmark_data_smithi102_461693_object15:head# 2024-04-25T09:26:19.440 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8aedfdc0:::benchmark_data_smithi102_461693_object193:head# 2024-04-25T09:26:19.440 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8afeae38:::benchmark_data_smithi102_461693_object221:head# 2024-04-25T09:26:19.441 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8b078005:::benchmark_data_smithi102_461693_object60:head# 2024-04-25T09:26:19.442 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8b4ee6c4:::benchmark_data_smithi102_461693_object166:head# 2024-04-25T09:26:19.443 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8c12de8e:::benchmark_data_smithi102_467209_object21:head# 2024-04-25T09:26:19.443 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8ca6a195:::benchmark_data_smithi102_461693_object172:head# 2024-04-25T09:26:19.444 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8cdff5e0:::benchmark_data_smithi102_461693_object17:head# 2024-04-25T09:26:19.444 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8d3cebfe:::benchmark_data_smithi102_467209_object47:head# 2024-04-25T09:26:19.445 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8d571cff:::benchmark_data_smithi102_461693_object255:head# 2024-04-25T09:26:19.445 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8eeaf510:::benchmark_data_smithi102_461693_object22:head# 2024-04-25T09:26:19.446 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8f3dbf2e:::benchmark_data_smithi102_461693_object110:head# 2024-04-25T09:26:19.446 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9036035d:::benchmark_data_smithi102_467209_object94:head# 2024-04-25T09:26:19.447 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:910c4b69:::benchmark_data_smithi102_467209_object17:head# 2024-04-25T09:26:19.447 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:91d0aef9:::benchmark_data_smithi102_461693_object132:head# 2024-04-25T09:26:19.448 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9269286e:::benchmark_data_smithi102_467209_object26:head# 2024-04-25T09:26:19.448 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:92c15cd6:::benchmark_data_smithi102_467209_object22:head# 2024-04-25T09:26:19.449 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:93bbd15e:::benchmark_data_smithi102_461693_object311:head# 2024-04-25T09:26:19.449 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:93cb41d3:::benchmark_data_smithi102_467209_object135:head# 2024-04-25T09:26:19.450 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:941e50ae:::benchmark_data_smithi102_467209_object113:head# 2024-04-25T09:26:19.450 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:942bc514:::benchmark_data_smithi102_461693_object77:head# 2024-04-25T09:26:19.451 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:94d0ab29:::benchmark_data_smithi102_461693_object285:head# 2024-04-25T09:26:19.451 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:94ec3b32:::benchmark_data_smithi102_461693_object72:head# 2024-04-25T09:26:19.452 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:95092f81:::benchmark_data_smithi102_461693_object220:head# 2024-04-25T09:26:19.452 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:95a9ddc7:::benchmark_data_smithi102_467209_object142:head# 2024-04-25T09:26:19.453 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:95b06a69:::benchmark_data_smithi102_461693_object67:head# 2024-04-25T09:26:19.453 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:95f571da:::benchmark_data_smithi102_467209_object66:head# 2024-04-25T09:26:19.453 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:969b0eab:::benchmark_data_smithi102_467209_object106:head# 2024-04-25T09:26:19.454 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:96ade7c2:::benchmark_data_smithi102_461693_object115:head# 2024-04-25T09:26:19.454 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:96e9fce0:::benchmark_data_smithi102_467209_object38:head# 2024-04-25T09:26:19.455 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:971cae83:::benchmark_data_smithi102_461693_object196:head# 2024-04-25T09:26:19.457 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:978b03f9:::benchmark_data_smithi102_461693_object236:head# 2024-04-25T09:26:19.457 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9862f725:::benchmark_data_smithi102_467209_object133:head# 2024-04-25T09:26:19.458 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:98701baa:::benchmark_data_smithi102_461693_object55:head# 2024-04-25T09:26:19.458 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:98e4a738:::benchmark_data_smithi102_461693_object127:head# 2024-04-25T09:26:19.459 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9aa9891c:::benchmark_data_smithi102_461693_object5:head# 2024-04-25T09:26:19.459 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9afe2884:::benchmark_data_smithi102_461693_object273:head# 2024-04-25T09:26:19.460 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9bd5d2e8:::benchmark_data_smithi102_461693_object265:head# 2024-04-25T09:26:19.460 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9bf31ddf:::benchmark_data_smithi102_461693_object83:head# 2024-04-25T09:26:19.461 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9c9b760d:::benchmark_data_smithi102_461693_object291:head# 2024-04-25T09:26:19.461 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9c9c7e9b:::benchmark_data_smithi102_461693_object70:head# 2024-04-25T09:26:19.461 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9cc82558:::benchmark_data_smithi102_461693_object108:head# 2024-04-25T09:26:19.462 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9d35491a:::benchmark_data_smithi102_467209_object158:head# 2024-04-25T09:26:19.462 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9d570aaf:::benchmark_data_smithi102_467209_object13:head# 2024-04-25T09:26:19.463 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9e60200b:::benchmark_data_smithi102_461693_object26:head# 2024-04-25T09:26:19.463 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9ebdc7a5:::benchmark_data_smithi102_461693_object18:head# 2024-04-25T09:26:19.464 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9eccdf33:::benchmark_data_smithi102_461693_object131:head# 2024-04-25T09:26:19.464 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9f958dac:::benchmark_data_smithi102_467209_object48:head# 2024-04-25T09:26:19.464 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9fcd1944:::benchmark_data_smithi102_461693_object198:head# 2024-04-25T09:26:19.465 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9fcee8c9:::benchmark_data_smithi102_461693_object263:head# 2024-04-25T09:26:19.465 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a109394d:::benchmark_data_smithi102_461693_object49:head# 2024-04-25T09:26:19.466 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a284d2b1:::benchmark_data_smithi102_461693_object40:head# 2024-04-25T09:26:19.466 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a2c77c71:::benchmark_data_smithi102_467209_object72:head# 2024-04-25T09:26:19.467 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a2ff1fbf:::benchmark_data_smithi102_461693_object52:head# 2024-04-25T09:26:19.467 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a32db852:::benchmark_data_smithi102_461693_object232:head# 2024-04-25T09:26:19.468 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a34ad8b3:::benchmark_data_smithi102_467209_object56:head# 2024-04-25T09:26:19.468 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a4f4d3a4:::benchmark_data_smithi102_461693_object261:head# 2024-04-25T09:26:19.469 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a7538180:::benchmark_data_smithi102_461693_object308:head# 2024-04-25T09:26:19.469 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:aa0c2ca6:::benchmark_data_smithi102_461693_object213:head# 2024-04-25T09:26:19.469 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:aa2b501d:::benchmark_data_smithi102_461693_object218:head# 2024-04-25T09:26:19.470 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:aa40adca:::benchmark_data_smithi102_467209_object24:head# 2024-04-25T09:26:19.470 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:aae249da:::benchmark_data_smithi102_461693_object182:head# 2024-04-25T09:26:19.471 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:aaf7ac98:::benchmark_data_smithi102_467209_object49:head# 2024-04-25T09:26:19.471 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ab0fe793:::benchmark_data_smithi102_467209_object159:head# 2024-04-25T09:26:19.472 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ab44e8cd:::benchmark_data_smithi102_467209_object86:head# 2024-04-25T09:26:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:abbc718e:::benchmark_data_smithi102_467209_object29:head# 2024-04-25T09:26:19.473 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac080002:::benchmark_data_smithi102_461693_object13:head# 2024-04-25T09:26:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac459348:::benchmark_data_smithi102_467209_object124:head# 2024-04-25T09:26:19.474 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac64182c:::benchmark_data_smithi102_461693_object295:head# 2024-04-25T09:26:19.475 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac73d9db:::benchmark_data_smithi102_467209_object95:head# 2024-04-25T09:26:19.475 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac96a035:::benchmark_data_smithi102_461693_object46:head# 2024-04-25T09:26:19.476 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:acd4c3d0:::benchmark_data_smithi102_461693_object226:head# 2024-04-25T09:26:19.476 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ad92f4ab:::benchmark_data_smithi102_461693_object303:head# 2024-04-25T09:26:19.477 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:adef340d:::benchmark_data_smithi102_461693_object61:head# 2024-04-25T09:26:19.477 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:afa99dfb:::benchmark_data_smithi102_467209_object67:head# 2024-04-25T09:26:19.478 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b1d35c85:::benchmark_data_smithi102_461693_object200:head# 2024-04-25T09:26:19.478 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b1d73141:::benchmark_data_smithi102_461693_object112:head# 2024-04-25T09:26:19.479 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b2c8f166:::benchmark_data_smithi102_467209_object46:head# 2024-04-25T09:26:19.479 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b4852218:::benchmark_data_smithi102_461693_object277:head# 2024-04-25T09:26:19.480 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b56c2919:::benchmark_data_smithi102_461693_object270:head# 2024-04-25T09:26:19.480 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b56e0ab2:::benchmark_data_smithi102_461693_object231:head# 2024-04-25T09:26:19.481 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b5c35fa0:::benchmark_data_smithi102_461693_object130:head# 2024-04-25T09:26:19.481 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b5d52fee:::benchmark_data_smithi102_467209_object36:head# 2024-04-25T09:26:19.481 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b79610bd:::benchmark_data_smithi102_467209_object150:head# 2024-04-25T09:26:19.482 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b7e1813e:::benchmark_data_smithi102_461693_object185:head# 2024-04-25T09:26:19.483 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:26:19.483 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b8772720:::benchmark_data_smithi102_461693_object289:head# 2024-04-25T09:26:19.483 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b925424a:::benchmark_data_smithi102_467209_object114:head# 2024-04-25T09:26:19.484 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bba685cf:::benchmark_data_smithi102_461693_object155:head# 2024-04-25T09:26:19.484 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bc0f76dd:::benchmark_data_smithi102_467209_object71:head# 2024-04-25T09:26:19.485 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bc36f44b:::benchmark_data_smithi102_461693_object296:head# 2024-04-25T09:26:19.485 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bc9448f0:::benchmark_data_smithi102_461693_object82:head# 2024-04-25T09:26:19.486 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bd20be31:::benchmark_data_smithi102_461693_object93:head# 2024-04-25T09:26:19.486 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bda0949f:::benchmark_data_smithi102_461693_object181:head# 2024-04-25T09:26:19.487 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bdd24e14:::benchmark_data_smithi102_467209_object73:head# 2024-04-25T09:26:19.487 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bde44e2f:::benchmark_data_smithi102_461693_object95:head# 2024-04-25T09:26:19.488 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:be6d30be:::benchmark_data_smithi102_461693_object207:head# 2024-04-25T09:26:19.488 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bf7281fd:::benchmark_data_smithi102_467209_object145:head# 2024-04-25T09:26:19.489 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bf7d046b:::benchmark_data_smithi102_461693_object278:head# 2024-04-25T09:26:19.489 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c1eaa85c:::benchmark_data_smithi102_461693_object137:head# 2024-04-25T09:26:19.490 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c2ae1d76:::benchmark_data_smithi102_467209_object141:head# 2024-04-25T09:26:19.490 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c2d6c8cd:::benchmark_data_smithi102_467209_object91:head# 2024-04-25T09:26:19.491 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c37f1115:::benchmark_data_smithi102_461693_object310:head# 2024-04-25T09:26:19.491 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c402eb86:::benchmark_data_smithi102_467209_object149:head# 2024-04-25T09:26:19.492 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c50db599:::benchmark_data_smithi102_461693_object176:head# 2024-04-25T09:26:19.492 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c53ef393:::benchmark_data_smithi102_461693_object279:head# 2024-04-25T09:26:19.493 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c53f44d8:::benchmark_data_smithi102_461693_object239:head# 2024-04-25T09:26:19.493 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c551c4c0:::benchmark_data_smithi102_461693_object123:head# 2024-04-25T09:26:19.493 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c55d9b9d:::benchmark_data_smithi102_461693_object237:head# 2024-04-25T09:26:19.494 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c5e9ecd2:::benchmark_data_smithi102_467209_object5:head# 2024-04-25T09:26:19.494 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c6ad85a0:::benchmark_data_smithi102_461693_object41:head# 2024-04-25T09:26:19.495 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c6d379b1:::benchmark_data_smithi102_461693_object304:head# 2024-04-25T09:26:19.495 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c74b1c18:::benchmark_data_smithi102_461693_object113:head# 2024-04-25T09:26:19.496 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c7aec9f5:::benchmark_data_smithi102_467209_object18:head# 2024-04-25T09:26:19.496 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c7f88b4e:::benchmark_data_smithi102_461693_object284:head# 2024-04-25T09:26:19.497 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c7fe9f67:::benchmark_data_smithi102_467209_object43:head# 2024-04-25T09:26:19.497 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c88cd082:::benchmark_data_smithi102_461693_object259:head# 2024-04-25T09:26:19.497 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c8e2e872:::benchmark_data_smithi102_461693_object39:head# 2024-04-25T09:26:19.498 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c93a1b98:::benchmark_data_smithi102_461693_object142:head# 2024-04-25T09:26:19.498 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c986488b:::benchmark_data_smithi102_467209_object4:head# 2024-04-25T09:26:19.499 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c9c320e5:::benchmark_data_smithi102_461693_object249:head# 2024-04-25T09:26:19.499 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ca288c06:::benchmark_data_smithi102_461693_object153:head# 2024-04-25T09:26:19.500 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ca29967a:::benchmark_data_smithi102_467209_object144:head# 2024-04-25T09:26:19.500 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:caf11a5f:::benchmark_data_smithi102_467209_object111:head# 2024-04-25T09:26:19.501 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cba55ca9:::benchmark_data_smithi102_461693_object283:head# 2024-04-25T09:26:19.502 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cbd30596:::benchmark_data_smithi102_461693_object145:head# 2024-04-25T09:26:19.502 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cc022f04:::benchmark_data_smithi102_467209_object98:head# 2024-04-25T09:26:19.503 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cc2c6f61:::benchmark_data_smithi102_467209_object108:head# 2024-04-25T09:26:19.503 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cc3765a4:::benchmark_data_smithi102_461693_object0:head# 2024-04-25T09:26:19.504 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ce561cbb:::benchmark_data_smithi102_461693_object244:head# 2024-04-25T09:26:19.504 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cec1e000:::benchmark_data_smithi102_461693_object171:head# 2024-04-25T09:26:19.505 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cefbcfec:::benchmark_data_smithi102_467209_object28:head# 2024-04-25T09:26:19.505 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d07784f4:::benchmark_data_smithi102_461693_object107:head# 2024-04-25T09:26:19.506 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d1040409:::benchmark_data_smithi102_467209_object119:head# 2024-04-25T09:26:19.506 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d131843b:::benchmark_data_smithi102_461693_object119:head# 2024-04-25T09:26:19.511 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d1c66f30:::benchmark_data_smithi102_467209_object152:head# 2024-04-25T09:26:19.511 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d1f2e35b:::benchmark_data_smithi102_461693_object257:head# 2024-04-25T09:26:19.511 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d28830cc:::benchmark_data_smithi102_467209_object116:head# 2024-04-25T09:26:19.511 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d2bb2fe0:::benchmark_data_smithi102_461693_object197:head# 2024-04-25T09:26:19.511 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d2f5066b:::benchmark_data_smithi102_461693_object307:head# 2024-04-25T09:26:19.511 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d39e771b:::benchmark_data_smithi102_467209_object138:head# 2024-04-25T09:26:19.511 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d3a5ef94:::benchmark_data_smithi102_467209_object62:head# 2024-04-25T09:26:19.511 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d4b957d9:::benchmark_data_smithi102_467209_object102:head# 2024-04-25T09:26:19.512 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d7474635:::benchmark_data_smithi102_467209_object35:head# 2024-04-25T09:26:19.512 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d77f7f03:::benchmark_data_smithi102_461693_object170:head# 2024-04-25T09:26:19.512 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d95e4094:::benchmark_data_smithi102_461693_object105:head# 2024-04-25T09:26:19.513 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:da23c414:::benchmark_data_smithi102_461693_object57:head# 2024-04-25T09:26:19.513 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:da5b1f76:::benchmark_data_smithi102_461693_object73:head# 2024-04-25T09:26:19.514 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:da5b6865:::benchmark_data_smithi102_461693_object290:head# 2024-04-25T09:26:19.514 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:db2f70fa:::benchmark_data_smithi102_461693_object230:head# 2024-04-25T09:26:19.515 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:db345364:::benchmark_data_smithi102_467209_object115:head# 2024-04-25T09:26:19.515 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:db42a2eb:::benchmark_data_smithi102_467209_object97:head# 2024-04-25T09:26:19.516 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dbf5b9d1:::benchmark_data_smithi102_461693_object312:head# 2024-04-25T09:26:19.516 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dcecc90c:::benchmark_data_smithi102_467209_object23:head# 2024-04-25T09:26:19.517 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:de452fc5:::benchmark_data_smithi102_461693_object167:head# 2024-04-25T09:26:19.517 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:de96a9bf:::benchmark_data_smithi102_467209_object27:head# 2024-04-25T09:26:19.517 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e195d889:::benchmark_data_smithi102_461693_object129:head# 2024-04-25T09:26:19.518 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e2323390:::benchmark_data_smithi102_461693_object33:head# 2024-04-25T09:26:19.518 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e2b7382e:::benchmark_data_smithi102_461693_object210:head# 2024-04-25T09:26:19.519 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e2bbe377:::benchmark_data_smithi102_461693_object114:head# 2024-04-25T09:26:19.519 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e301fcee:::benchmark_data_smithi102_461693_object62:head# 2024-04-25T09:26:19.520 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e302dfe1:::benchmark_data_smithi102_461693_object227:head# 2024-04-25T09:26:19.520 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e4324c30:::benchmark_data_smithi102_461693_object158:head# 2024-04-25T09:26:19.521 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e4358590:::benchmark_data_smithi102_467209_object104:head# 2024-04-25T09:26:19.521 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e447aef9:::benchmark_data_smithi102_461693_object180:head# 2024-04-25T09:26:19.522 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e4558303:::benchmark_data_smithi102_461693_object4:head# 2024-04-25T09:26:19.522 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e5053873:::benchmark_data_smithi102_461693_object81:head# 2024-04-25T09:26:19.523 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e5210074:::benchmark_data_smithi102_461693_object3:head# 2024-04-25T09:26:19.523 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e6300266:::benchmark_data_smithi102_467209_object41:head# 2024-04-25T09:26:19.524 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e7382a9c:::benchmark_data_smithi102_461693_object299:head# 2024-04-25T09:26:19.524 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e7710cbe:::benchmark_data_smithi102_467209_object53:head# 2024-04-25T09:26:19.525 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e819bcc3:::benchmark_data_smithi102_461693_object84:head# 2024-04-25T09:26:19.525 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e84e09f2:::benchmark_data_smithi102_467209_object64:head# 2024-04-25T09:26:19.526 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e88ddb38:::benchmark_data_smithi102_467209_object128:head# 2024-04-25T09:26:19.526 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e8d6bf29:::benchmark_data_smithi102_461693_object101:head# 2024-04-25T09:26:19.527 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e9bbbf61:::benchmark_data_smithi102_467209_object93:head# 2024-04-25T09:26:19.527 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ea24f1cb:::benchmark_data_smithi102_461693_object186:head# 2024-04-25T09:26:19.528 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ec9bc09f:::benchmark_data_smithi102_461693_object229:head# 2024-04-25T09:26:19.528 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ecca2721:::benchmark_data_smithi102_461693_object134:head# 2024-04-25T09:26:19.529 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ecdd65cb:::benchmark_data_smithi102_461693_object269:head# 2024-04-25T09:26:19.529 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ed51570a:::benchmark_data_smithi102_467209_object80:head# 2024-04-25T09:26:19.530 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ee2ac7ee:::benchmark_data_smithi102_467209_object10:head# 2024-04-25T09:26:19.530 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ee986a84:::benchmark_data_smithi102_467209_object117:head# 2024-04-25T09:26:19.531 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:eed87a51:::benchmark_data_smithi102_461693_object191:head# 2024-04-25T09:26:19.531 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ef8637ba:::benchmark_data_smithi102_461693_object151:head# 2024-04-25T09:26:19.532 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f02b8d22:::benchmark_data_smithi102_461693_object168:head# 2024-04-25T09:26:19.532 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f06a39d9:::benchmark_data_smithi102_461693_object275:head# 2024-04-25T09:26:19.533 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f136e2f9:::benchmark_data_smithi102_461693_object267:head# 2024-04-25T09:26:19.533 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f19ac1c3:::benchmark_data_smithi102_467209_object31:head# 2024-04-25T09:26:19.534 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f2467763:::benchmark_data_smithi102_461693_object156:head# 2024-04-25T09:26:19.534 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f2e23aef:::benchmark_data_smithi102_467209_object33:head# 2024-04-25T09:26:19.535 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f313055e:::benchmark_data_smithi102_467209_object103:head# 2024-04-25T09:26:19.535 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f3685b12:::benchmark_data_smithi102_461693_object212:head# 2024-04-25T09:26:19.536 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f39374a5:::benchmark_data_smithi102_461693_object184:head# 2024-04-25T09:26:19.536 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f5c19b84:::benchmark_data_smithi102_461693_object225:head# 2024-04-25T09:26:19.536 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f5fbc496:::benchmark_data_smithi102_467209_object96:head# 2024-04-25T09:26:19.537 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f69416ea:::benchmark_data_smithi102_461693_object235:head# 2024-04-25T09:26:19.537 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f6d72824:::benchmark_data_smithi102_467209_object140:head# 2024-04-25T09:26:19.538 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f7295629:::benchmark_data_smithi102_461693_object87:head# 2024-04-25T09:26:19.538 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f8aeba6c:::benchmark_data_smithi102_467209_object148:head# 2024-04-25T09:26:19.539 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f90d383f:::benchmark_data_smithi102_461693_object203:head# 2024-04-25T09:26:19.539 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f939e8fc:::benchmark_data_smithi102_461693_object48:head# 2024-04-25T09:26:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f993a43e:::benchmark_data_smithi102_461693_object99:head# 2024-04-25T09:26:19.540 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f9fcdfe5:::benchmark_data_smithi102_467209_object52:head# 2024-04-25T09:26:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f9fdbd9c:::benchmark_data_smithi102_461693_object86:head# 2024-04-25T09:26:19.541 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fa2a1c9b:::benchmark_data_smithi102_461693_object47:head# 2024-04-25T09:26:19.542 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:facfae24:::benchmark_data_smithi102_467209_object89:head# 2024-04-25T09:26:19.542 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fb939772:::benchmark_data_smithi102_461693_object9:head# 2024-04-25T09:26:19.543 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fbd71e78:::benchmark_data_smithi102_467209_object90:head# 2024-04-25T09:26:19.543 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fca3284e:::benchmark_data_smithi102_461693_object187:head# 2024-04-25T09:26:19.544 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fe98a8ca:::benchmark_data_smithi102_461693_object258:head# 2024-04-25T09:26:19.544 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ff7040e0:::benchmark_data_smithi102_461693_object281:head# 2024-04-25T09:26:19.544 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ffab2f9e:::benchmark_data_smithi102_461693_object154:head# 2024-04-25T09:26:19.545 INFO:tasks.workunit.client.0.smithi102.stderr:Export successful 2024-04-25T09:26:19.545 INFO:tasks.workunit.client.0.smithi102.stdout: marking collection for removal 2024-04-25T09:26:19.545 INFO:tasks.workunit.client.0.smithi102.stdout:setting '_remove' omap key 2024-04-25T09:26:19.546 INFO:tasks.workunit.client.0.smithi102.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:26:20.432 INFO:tasks.workunit.client.0.smithi102.stderr:Remove successful 2024-04-25T09:26:20.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:143: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:26:21.586 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:26:21.586 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.1 2024-04-25T09:26:21.587 INFO:tasks.workunit.client.0.smithi102.stderr:PG export's map 10 is older than OSD's oldest_map 61 2024-04-25T09:26:21.587 INFO:tasks.workunit.client.0.smithi102.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:26:22.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:144: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:26:22.589 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:26:22.589 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.0 2024-04-25T09:26:22.589 INFO:tasks.workunit.client.0.smithi102.stderr:PG export's map 10 is older than OSD's oldest_map 61 2024-04-25T09:26:22.589 INFO:tasks.workunit.client.0.smithi102.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:26:23.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:147: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-04-25T09:26:23.668 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:26:23.668 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.1 2024-04-25T09:26:23.669 INFO:tasks.workunit.client.0.smithi102.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:26:23.669 INFO:tasks.workunit.client.0.smithi102.stderr:PG export's map 10 is older than OSD's oldest_map 61 2024-04-25T09:26:23.670 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:80decefa:::benchmark_data_smithi102_461693_object215:head# 2024-04-25T09:26:23.670 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.709 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:82504851:::benchmark_data_smithi102_461693_object59:head# 2024-04-25T09:26:23.709 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.727 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:82bc124c:::benchmark_data_smithi102_461693_object242:head# 2024-04-25T09:26:23.727 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.743 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:883cb801:::benchmark_data_smithi102_461693_object150:head# 2024-04-25T09:26:23.743 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.760 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:889e9aa5:::benchmark_data_smithi102_461693_object250:head# 2024-04-25T09:26:23.760 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.776 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:890d9aaf:::benchmark_data_smithi102_461693_object96:head# 2024-04-25T09:26:23.776 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.793 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8a093910:::benchmark_data_smithi102_461693_object15:head# 2024-04-25T09:26:23.793 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.809 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8aedfdc0:::benchmark_data_smithi102_461693_object193:head# 2024-04-25T09:26:23.809 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.826 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8afeae38:::benchmark_data_smithi102_461693_object221:head# 2024-04-25T09:26:23.826 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.843 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8b078005:::benchmark_data_smithi102_461693_object60:head# 2024-04-25T09:26:23.843 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.859 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8b4ee6c4:::benchmark_data_smithi102_461693_object166:head# 2024-04-25T09:26:23.860 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.876 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8ca6a195:::benchmark_data_smithi102_461693_object172:head# 2024-04-25T09:26:23.876 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.893 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8cdff5e0:::benchmark_data_smithi102_461693_object17:head# 2024-04-25T09:26:23.893 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.911 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8d571cff:::benchmark_data_smithi102_461693_object255:head# 2024-04-25T09:26:23.912 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.926 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8eeaf510:::benchmark_data_smithi102_461693_object22:head# 2024-04-25T09:26:23.926 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.943 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8f3dbf2e:::benchmark_data_smithi102_461693_object110:head# 2024-04-25T09:26:23.943 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.960 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:91d0aef9:::benchmark_data_smithi102_461693_object132:head# 2024-04-25T09:26:23.960 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.976 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:93bbd15e:::benchmark_data_smithi102_461693_object311:head# 2024-04-25T09:26:23.977 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:23.993 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:942bc514:::benchmark_data_smithi102_461693_object77:head# 2024-04-25T09:26:23.993 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.010 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:94d0ab29:::benchmark_data_smithi102_461693_object285:head# 2024-04-25T09:26:24.010 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.027 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:94ec3b32:::benchmark_data_smithi102_461693_object72:head# 2024-04-25T09:26:24.027 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.043 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:95092f81:::benchmark_data_smithi102_461693_object220:head# 2024-04-25T09:26:24.043 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.060 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:95b06a69:::benchmark_data_smithi102_461693_object67:head# 2024-04-25T09:26:24.060 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.076 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:96ade7c2:::benchmark_data_smithi102_461693_object115:head# 2024-04-25T09:26:24.076 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.093 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:971cae83:::benchmark_data_smithi102_461693_object196:head# 2024-04-25T09:26:24.093 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.109 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:978b03f9:::benchmark_data_smithi102_461693_object236:head# 2024-04-25T09:26:24.110 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.167 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:98701baa:::benchmark_data_smithi102_461693_object55:head# 2024-04-25T09:26:24.167 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.210 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:98e4a738:::benchmark_data_smithi102_461693_object127:head# 2024-04-25T09:26:24.210 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.252 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9aa9891c:::benchmark_data_smithi102_461693_object5:head# 2024-04-25T09:26:24.252 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.268 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9afe2884:::benchmark_data_smithi102_461693_object273:head# 2024-04-25T09:26:24.268 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.285 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9bd5d2e8:::benchmark_data_smithi102_461693_object265:head# 2024-04-25T09:26:24.285 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.301 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9bf31ddf:::benchmark_data_smithi102_461693_object83:head# 2024-04-25T09:26:24.301 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.318 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9c9b760d:::benchmark_data_smithi102_461693_object291:head# 2024-04-25T09:26:24.318 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.335 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9c9c7e9b:::benchmark_data_smithi102_461693_object70:head# 2024-04-25T09:26:24.335 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.352 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9cc82558:::benchmark_data_smithi102_461693_object108:head# 2024-04-25T09:26:24.352 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.368 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9e60200b:::benchmark_data_smithi102_461693_object26:head# 2024-04-25T09:26:24.368 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.385 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9ebdc7a5:::benchmark_data_smithi102_461693_object18:head# 2024-04-25T09:26:24.385 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.402 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9eccdf33:::benchmark_data_smithi102_461693_object131:head# 2024-04-25T09:26:24.402 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.422 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9fcd1944:::benchmark_data_smithi102_461693_object198:head# 2024-04-25T09:26:24.422 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.435 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9fcee8c9:::benchmark_data_smithi102_461693_object263:head# 2024-04-25T09:26:24.435 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.452 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a109394d:::benchmark_data_smithi102_461693_object49:head# 2024-04-25T09:26:24.452 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.468 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a284d2b1:::benchmark_data_smithi102_461693_object40:head# 2024-04-25T09:26:24.469 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.485 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a2ff1fbf:::benchmark_data_smithi102_461693_object52:head# 2024-04-25T09:26:24.485 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.502 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a32db852:::benchmark_data_smithi102_461693_object232:head# 2024-04-25T09:26:24.502 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.518 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a4f4d3a4:::benchmark_data_smithi102_461693_object261:head# 2024-04-25T09:26:24.518 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.535 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a7538180:::benchmark_data_smithi102_461693_object308:head# 2024-04-25T09:26:24.535 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.552 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aa0c2ca6:::benchmark_data_smithi102_461693_object213:head# 2024-04-25T09:26:24.552 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.569 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aa2b501d:::benchmark_data_smithi102_461693_object218:head# 2024-04-25T09:26:24.569 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.585 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aae249da:::benchmark_data_smithi102_461693_object182:head# 2024-04-25T09:26:24.585 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.602 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac080002:::benchmark_data_smithi102_461693_object13:head# 2024-04-25T09:26:24.602 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.619 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac64182c:::benchmark_data_smithi102_461693_object295:head# 2024-04-25T09:26:24.619 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.635 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac96a035:::benchmark_data_smithi102_461693_object46:head# 2024-04-25T09:26:24.635 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.652 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:acd4c3d0:::benchmark_data_smithi102_461693_object226:head# 2024-04-25T09:26:24.652 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.668 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ad92f4ab:::benchmark_data_smithi102_461693_object303:head# 2024-04-25T09:26:24.669 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.685 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:adef340d:::benchmark_data_smithi102_461693_object61:head# 2024-04-25T09:26:24.685 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.702 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b1d35c85:::benchmark_data_smithi102_461693_object200:head# 2024-04-25T09:26:24.702 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.719 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b1d73141:::benchmark_data_smithi102_461693_object112:head# 2024-04-25T09:26:24.719 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.735 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b4852218:::benchmark_data_smithi102_461693_object277:head# 2024-04-25T09:26:24.735 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.752 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b56c2919:::benchmark_data_smithi102_461693_object270:head# 2024-04-25T09:26:24.752 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.769 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b56e0ab2:::benchmark_data_smithi102_461693_object231:head# 2024-04-25T09:26:24.769 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.785 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b5c35fa0:::benchmark_data_smithi102_461693_object130:head# 2024-04-25T09:26:24.786 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.802 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b7e1813e:::benchmark_data_smithi102_461693_object185:head# 2024-04-25T09:26:24.802 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.819 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:26:24.819 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.835 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b8772720:::benchmark_data_smithi102_461693_object289:head# 2024-04-25T09:26:24.836 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.852 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bba685cf:::benchmark_data_smithi102_461693_object155:head# 2024-04-25T09:26:24.852 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.894 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc36f44b:::benchmark_data_smithi102_461693_object296:head# 2024-04-25T09:26:24.894 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.911 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc9448f0:::benchmark_data_smithi102_461693_object82:head# 2024-04-25T09:26:24.911 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.927 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bd20be31:::benchmark_data_smithi102_461693_object93:head# 2024-04-25T09:26:24.928 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.944 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bda0949f:::benchmark_data_smithi102_461693_object181:head# 2024-04-25T09:26:24.944 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.961 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bde44e2f:::benchmark_data_smithi102_461693_object95:head# 2024-04-25T09:26:24.961 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.977 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:be6d30be:::benchmark_data_smithi102_461693_object207:head# 2024-04-25T09:26:24.978 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:24.994 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bf7d046b:::benchmark_data_smithi102_461693_object278:head# 2024-04-25T09:26:24.994 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.035 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c1eaa85c:::benchmark_data_smithi102_461693_object137:head# 2024-04-25T09:26:25.035 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.086 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c37f1115:::benchmark_data_smithi102_461693_object310:head# 2024-04-25T09:26:25.086 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.151 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c50db599:::benchmark_data_smithi102_461693_object176:head# 2024-04-25T09:26:25.151 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.193 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c53ef393:::benchmark_data_smithi102_461693_object279:head# 2024-04-25T09:26:25.193 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.211 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c53f44d8:::benchmark_data_smithi102_461693_object239:head# 2024-04-25T09:26:25.211 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.251 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c551c4c0:::benchmark_data_smithi102_461693_object123:head# 2024-04-25T09:26:25.251 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.308 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c55d9b9d:::benchmark_data_smithi102_461693_object237:head# 2024-04-25T09:26:25.308 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.344 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c6ad85a0:::benchmark_data_smithi102_461693_object41:head# 2024-04-25T09:26:25.345 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.361 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c6d379b1:::benchmark_data_smithi102_461693_object304:head# 2024-04-25T09:26:25.361 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.377 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c74b1c18:::benchmark_data_smithi102_461693_object113:head# 2024-04-25T09:26:25.378 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.394 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c7f88b4e:::benchmark_data_smithi102_461693_object284:head# 2024-04-25T09:26:25.394 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.411 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c88cd082:::benchmark_data_smithi102_461693_object259:head# 2024-04-25T09:26:25.411 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.427 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c8e2e872:::benchmark_data_smithi102_461693_object39:head# 2024-04-25T09:26:25.428 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.444 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c93a1b98:::benchmark_data_smithi102_461693_object142:head# 2024-04-25T09:26:25.444 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.461 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c9c320e5:::benchmark_data_smithi102_461693_object249:head# 2024-04-25T09:26:25.461 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.478 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ca288c06:::benchmark_data_smithi102_461693_object153:head# 2024-04-25T09:26:25.478 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.494 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cba55ca9:::benchmark_data_smithi102_461693_object283:head# 2024-04-25T09:26:25.495 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.511 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cbd30596:::benchmark_data_smithi102_461693_object145:head# 2024-04-25T09:26:25.511 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.528 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cc3765a4:::benchmark_data_smithi102_461693_object0:head# 2024-04-25T09:26:25.528 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.544 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ce561cbb:::benchmark_data_smithi102_461693_object244:head# 2024-04-25T09:26:25.544 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.561 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cec1e000:::benchmark_data_smithi102_461693_object171:head# 2024-04-25T09:26:25.561 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.578 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d07784f4:::benchmark_data_smithi102_461693_object107:head# 2024-04-25T09:26:25.578 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.594 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d131843b:::benchmark_data_smithi102_461693_object119:head# 2024-04-25T09:26:25.595 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.611 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d1f2e35b:::benchmark_data_smithi102_461693_object257:head# 2024-04-25T09:26:25.611 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.677 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d2bb2fe0:::benchmark_data_smithi102_461693_object197:head# 2024-04-25T09:26:25.677 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.719 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d2f5066b:::benchmark_data_smithi102_461693_object307:head# 2024-04-25T09:26:25.719 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.745 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d77f7f03:::benchmark_data_smithi102_461693_object170:head# 2024-04-25T09:26:25.745 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.761 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d95e4094:::benchmark_data_smithi102_461693_object105:head# 2024-04-25T09:26:25.761 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.778 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:da23c414:::benchmark_data_smithi102_461693_object57:head# 2024-04-25T09:26:25.778 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.794 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:da5b1f76:::benchmark_data_smithi102_461693_object73:head# 2024-04-25T09:26:25.795 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.811 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:da5b6865:::benchmark_data_smithi102_461693_object290:head# 2024-04-25T09:26:25.811 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.828 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:db2f70fa:::benchmark_data_smithi102_461693_object230:head# 2024-04-25T09:26:25.828 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.845 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dbf5b9d1:::benchmark_data_smithi102_461693_object312:head# 2024-04-25T09:26:25.845 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.861 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:de452fc5:::benchmark_data_smithi102_461693_object167:head# 2024-04-25T09:26:25.862 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.878 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e195d889:::benchmark_data_smithi102_461693_object129:head# 2024-04-25T09:26:25.878 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.895 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e2323390:::benchmark_data_smithi102_461693_object33:head# 2024-04-25T09:26:25.895 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.911 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e2b7382e:::benchmark_data_smithi102_461693_object210:head# 2024-04-25T09:26:25.912 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.928 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e2bbe377:::benchmark_data_smithi102_461693_object114:head# 2024-04-25T09:26:25.928 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.945 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e301fcee:::benchmark_data_smithi102_461693_object62:head# 2024-04-25T09:26:25.945 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.961 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e302dfe1:::benchmark_data_smithi102_461693_object227:head# 2024-04-25T09:26:25.962 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.978 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e4324c30:::benchmark_data_smithi102_461693_object158:head# 2024-04-25T09:26:25.978 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:25.995 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e447aef9:::benchmark_data_smithi102_461693_object180:head# 2024-04-25T09:26:25.995 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.011 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e4558303:::benchmark_data_smithi102_461693_object4:head# 2024-04-25T09:26:26.012 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.028 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e5053873:::benchmark_data_smithi102_461693_object81:head# 2024-04-25T09:26:26.028 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.045 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e5210074:::benchmark_data_smithi102_461693_object3:head# 2024-04-25T09:26:26.045 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.062 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e7382a9c:::benchmark_data_smithi102_461693_object299:head# 2024-04-25T09:26:26.062 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.078 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e819bcc3:::benchmark_data_smithi102_461693_object84:head# 2024-04-25T09:26:26.078 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.095 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e8d6bf29:::benchmark_data_smithi102_461693_object101:head# 2024-04-25T09:26:26.095 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.111 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ea24f1cb:::benchmark_data_smithi102_461693_object186:head# 2024-04-25T09:26:26.112 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.162 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ec9bc09f:::benchmark_data_smithi102_461693_object229:head# 2024-04-25T09:26:26.162 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.203 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ecca2721:::benchmark_data_smithi102_461693_object134:head# 2024-04-25T09:26:26.203 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.254 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ecdd65cb:::benchmark_data_smithi102_461693_object269:head# 2024-04-25T09:26:26.254 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.287 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:eed87a51:::benchmark_data_smithi102_461693_object191:head# 2024-04-25T09:26:26.287 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.303 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ef8637ba:::benchmark_data_smithi102_461693_object151:head# 2024-04-25T09:26:26.303 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.320 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f02b8d22:::benchmark_data_smithi102_461693_object168:head# 2024-04-25T09:26:26.320 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.337 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f06a39d9:::benchmark_data_smithi102_461693_object275:head# 2024-04-25T09:26:26.337 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.353 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f136e2f9:::benchmark_data_smithi102_461693_object267:head# 2024-04-25T09:26:26.354 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.412 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f2467763:::benchmark_data_smithi102_461693_object156:head# 2024-04-25T09:26:26.412 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.428 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f3685b12:::benchmark_data_smithi102_461693_object212:head# 2024-04-25T09:26:26.429 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.445 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f39374a5:::benchmark_data_smithi102_461693_object184:head# 2024-04-25T09:26:26.445 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.462 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f5c19b84:::benchmark_data_smithi102_461693_object225:head# 2024-04-25T09:26:26.462 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.478 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f69416ea:::benchmark_data_smithi102_461693_object235:head# 2024-04-25T09:26:26.479 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.495 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f7295629:::benchmark_data_smithi102_461693_object87:head# 2024-04-25T09:26:26.495 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.512 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f90d383f:::benchmark_data_smithi102_461693_object203:head# 2024-04-25T09:26:26.512 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.529 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f939e8fc:::benchmark_data_smithi102_461693_object48:head# 2024-04-25T09:26:26.529 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.545 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f993a43e:::benchmark_data_smithi102_461693_object99:head# 2024-04-25T09:26:26.546 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.562 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f9fdbd9c:::benchmark_data_smithi102_461693_object86:head# 2024-04-25T09:26:26.562 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.579 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fa2a1c9b:::benchmark_data_smithi102_461693_object47:head# 2024-04-25T09:26:26.579 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.595 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fb939772:::benchmark_data_smithi102_461693_object9:head# 2024-04-25T09:26:26.595 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.637 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fca3284e:::benchmark_data_smithi102_461693_object187:head# 2024-04-25T09:26:26.637 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.654 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fe98a8ca:::benchmark_data_smithi102_461693_object258:head# 2024-04-25T09:26:26.654 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.670 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ff7040e0:::benchmark_data_smithi102_461693_object281:head# 2024-04-25T09:26:26.670 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.687 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ffab2f9e:::benchmark_data_smithi102_461693_object154:head# 2024-04-25T09:26:26.687 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:26.704 INFO:tasks.workunit.client.0.smithi102.stdout:write_pg epoch 10 info 1.1( v 10'145 (0'0,10'145] local-lis/les=7/8 n=145 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:26:26.721 INFO:tasks.workunit.client.0.smithi102.stdout:Import successful 2024-04-25T09:26:27.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:148: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:26:27.959 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:26:27.959 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.0 2024-04-25T09:26:27.960 INFO:tasks.workunit.client.0.smithi102.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:26:27.960 INFO:tasks.workunit.client.0.smithi102.stderr:PG export's map 10 is older than OSD's oldest_map 61 2024-04-25T09:26:27.960 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:000b3b92:::benchmark_data_smithi102_461693_object262:head# 2024-04-25T09:26:27.960 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:27.996 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:00374c2e:::benchmark_data_smithi102_461693_object216:head# 2024-04-25T09:26:27.996 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.021 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0301c80c:::benchmark_data_smithi102_461693_object11:head# 2024-04-25T09:26:28.022 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.033 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:036651e6:::benchmark_data_smithi102_461693_object111:head# 2024-04-25T09:26:28.033 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.059 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0402916e:::benchmark_data_smithi102_461693_object183:head# 2024-04-25T09:26:28.059 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.080 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:046b3940:::benchmark_data_smithi102_461693_object104:head# 2024-04-25T09:26:28.080 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.105 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0540312f:::benchmark_data_smithi102_461693_object69:head# 2024-04-25T09:26:28.105 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.163 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:054ee2e1:::benchmark_data_smithi102_461693_object300:head# 2024-04-25T09:26:28.164 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.214 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:069b73e0:::benchmark_data_smithi102_461693_object195:head# 2024-04-25T09:26:28.214 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.273 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:07526ebf:::benchmark_data_smithi102_461693_object248:head# 2024-04-25T09:26:28.274 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.303 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:08aeff1a:::benchmark_data_smithi102_461693_object43:head# 2024-04-25T09:26:28.303 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.318 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:09569df1:::benchmark_data_smithi102_461693_object305:head# 2024-04-25T09:26:28.318 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.339 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:099020c4:::benchmark_data_smithi102_461693_object152:head# 2024-04-25T09:26:28.339 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.364 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0a47cb3d:::benchmark_data_smithi102_461693_object266:head# 2024-04-25T09:26:28.364 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.389 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0a7de0f7:::benchmark_data_smithi102_461693_object208:head# 2024-04-25T09:26:28.389 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.414 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0b6a70be:::benchmark_data_smithi102_461693_object85:head# 2024-04-25T09:26:28.414 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.434 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0cc78a03:::benchmark_data_smithi102_461693_object21:head# 2024-04-25T09:26:28.434 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.455 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0cd9cd5d:::benchmark_data_smithi102_461693_object160:head# 2024-04-25T09:26:28.455 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.480 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0deb35d3:::benchmark_data_smithi102_461693_object217:head# 2024-04-25T09:26:28.480 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.489 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0e8b2dd0:::benchmark_data_smithi102_461693_object148:head# 2024-04-25T09:26:28.489 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.506 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0f24b073:::benchmark_data_smithi102_461693_object317:head# 2024-04-25T09:26:28.507 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.522 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:10af48fb:::benchmark_data_smithi102_461693_object240:head# 2024-04-25T09:26:28.522 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.539 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:11692bdb:::benchmark_data_smithi102_461693_object27:head# 2024-04-25T09:26:28.539 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.547 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:128e9745:::benchmark_data_smithi102_461693_object314:head# 2024-04-25T09:26:28.547 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.564 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:132fbbae:::benchmark_data_smithi102_461693_object92:head# 2024-04-25T09:26:28.564 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.572 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:14a6a0cd:::benchmark_data_smithi102_461693_object88:head# 2024-04-25T09:26:28.572 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.589 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1512b06b:::benchmark_data_smithi102_461693_object272:head# 2024-04-25T09:26:28.589 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.597 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1629e659:::benchmark_data_smithi102_461693_object256:head# 2024-04-25T09:26:28.597 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.605 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:16f7bd8c:::benchmark_data_smithi102_461693_object64:head# 2024-04-25T09:26:28.606 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.614 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1786312c:::benchmark_data_smithi102_461693_object37:head# 2024-04-25T09:26:28.614 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.631 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:19fa802a:::benchmark_data_smithi102_461693_object177:head# 2024-04-25T09:26:28.631 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.639 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1d2fbc72:::benchmark_data_smithi102_461693_object253:head# 2024-04-25T09:26:28.639 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.656 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1dbe156c:::benchmark_data_smithi102_461693_object301:head# 2024-04-25T09:26:28.656 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.664 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1e43a575:::benchmark_data_smithi102_461693_object109:head# 2024-04-25T09:26:28.664 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.681 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1e98b968:::benchmark_data_smithi102_461693_object8:head# 2024-04-25T09:26:28.681 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.698 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1e9976c2:::benchmark_data_smithi102_461693_object31:head# 2024-04-25T09:26:28.698 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.706 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:20ac13d7:::benchmark_data_smithi102_461693_object319:head# 2024-04-25T09:26:28.706 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.714 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2279d77a:::benchmark_data_smithi102_461693_object157:head# 2024-04-25T09:26:28.714 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.722 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:22b40e23:::benchmark_data_smithi102_461693_object233:head# 2024-04-25T09:26:28.723 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.731 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:23111692:::benchmark_data_smithi102_461693_object14:head# 2024-04-25T09:26:28.731 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.748 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:23426916:::benchmark_data_smithi102_461693_object164:head# 2024-04-25T09:26:28.748 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.756 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:238a3ae3:::benchmark_data_smithi102_461693_object211:head# 2024-04-25T09:26:28.756 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.772 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:239e27b3:::benchmark_data_smithi102_461693_object219:head# 2024-04-25T09:26:28.773 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.781 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:23b4b493:::benchmark_data_smithi102_461693_object192:head# 2024-04-25T09:26:28.781 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.789 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:24e58257:::benchmark_data_smithi102_461693_object141:head# 2024-04-25T09:26:28.789 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.798 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:24e6a109:::benchmark_data_smithi102_461693_object106:head# 2024-04-25T09:26:28.798 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.806 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:25997359:::benchmark_data_smithi102_461693_object264:head# 2024-04-25T09:26:28.806 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.814 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:25b16926:::benchmark_data_smithi102_461693_object29:head# 2024-04-25T09:26:28.814 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.823 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:26824afa:::benchmark_data_smithi102_461693_object94:head# 2024-04-25T09:26:28.823 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.831 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:26a55470:::benchmark_data_smithi102_461693_object274:head# 2024-04-25T09:26:28.831 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.848 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:272eb2c9:::benchmark_data_smithi102_461693_object149:head# 2024-04-25T09:26:28.848 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.856 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:273627fb:::benchmark_data_smithi102_461693_object125:head# 2024-04-25T09:26:28.856 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.864 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:273b3e12:::benchmark_data_smithi102_461693_object161:head# 2024-04-25T09:26:28.864 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.873 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:275d130a:::benchmark_data_smithi102_461693_object20:head# 2024-04-25T09:26:28.873 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.881 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:28694220:::benchmark_data_smithi102_461693_object136:head# 2024-04-25T09:26:28.881 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.889 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2a0bbe58:::benchmark_data_smithi102_461693_object199:head# 2024-04-25T09:26:28.890 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.898 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2afc2f48:::benchmark_data_smithi102_461693_object209:head# 2024-04-25T09:26:28.898 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.906 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2bbb9370:::benchmark_data_smithi102_461693_object287:head# 2024-04-25T09:26:28.906 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.914 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2bd520cd:::benchmark_data_smithi102_461693_object19:head# 2024-04-25T09:26:28.915 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.923 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2be202af:::benchmark_data_smithi102_461693_object234:head# 2024-04-25T09:26:28.923 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.931 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2c0b4651:::benchmark_data_smithi102_461693_object74:head# 2024-04-25T09:26:28.931 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.940 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2c9a727c:::benchmark_data_smithi102_461693_object165:head# 2024-04-25T09:26:28.940 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.948 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2d3d0890:::benchmark_data_smithi102_461693_object50:head# 2024-04-25T09:26:28.948 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.956 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2d3fef0a:::benchmark_data_smithi102_461693_object173:head# 2024-04-25T09:26:28.956 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.964 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2d4d8992:::benchmark_data_smithi102_461693_object53:head# 2024-04-25T09:26:28.965 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:28.998 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2ee095c4:::benchmark_data_smithi102_461693_object58:head# 2024-04-25T09:26:28.998 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.006 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2fd07b30:::benchmark_data_smithi102_461693_object194:head# 2024-04-25T09:26:29.006 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.015 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2fee0198:::benchmark_data_smithi102_461693_object2:head# 2024-04-25T09:26:29.015 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.023 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:34e9f16c:::benchmark_data_smithi102_461693_object260:head# 2024-04-25T09:26:29.023 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.031 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:355453a0:::benchmark_data_smithi102_461693_object118:head# 2024-04-25T09:26:29.031 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.040 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:363cd543:::benchmark_data_smithi102_461693_object34:head# 2024-04-25T09:26:29.040 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.048 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:36551f3c:::benchmark_data_smithi102_461693_object51:head# 2024-04-25T09:26:29.048 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.056 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3693711a:::benchmark_data_smithi102_461693_object306:head# 2024-04-25T09:26:29.056 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.065 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:37225bcc:::benchmark_data_smithi102_461693_object117:head# 2024-04-25T09:26:29.065 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.073 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:37c0cca3:::benchmark_data_smithi102_461693_object282:head# 2024-04-25T09:26:29.073 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.082 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:389b2f68:::benchmark_data_smithi102_461693_object54:head# 2024-04-25T09:26:29.082 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.090 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:38f5ec38:::benchmark_data_smithi102_461693_object12:head# 2024-04-25T09:26:29.090 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.098 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:39a2e424:::benchmark_data_smithi102_461693_object16:head# 2024-04-25T09:26:29.098 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.107 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3b72c6d5:::benchmark_data_smithi102_461693_object28:head# 2024-04-25T09:26:29.107 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.115 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3bca4278:::benchmark_data_smithi102_461693_object6:head# 2024-04-25T09:26:29.115 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.123 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3c78f5d8:::benchmark_data_smithi102_461693_object124:head# 2024-04-25T09:26:29.123 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.132 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3ccb0f2b:::benchmark_data_smithi102_461693_object144:head# 2024-04-25T09:26:29.132 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.140 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3d133806:::benchmark_data_smithi102_461693_object243:head# 2024-04-25T09:26:29.140 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.148 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3d2ad51e:::benchmark_data_smithi102_461693_object38:head# 2024-04-25T09:26:29.149 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.157 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3d611f0a:::benchmark_data_smithi102_461693_object32:head# 2024-04-25T09:26:29.157 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.165 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3f6d6ad9:::benchmark_data_smithi102_461693_object204:head# 2024-04-25T09:26:29.165 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.173 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3f7d2427:::benchmark_data_smithi102_461693_object205:head# 2024-04-25T09:26:29.173 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.182 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:40581110:::benchmark_data_smithi102_461693_object147:head# 2024-04-25T09:26:29.182 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.190 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4066b8f9:::benchmark_data_smithi102_461693_object103:head# 2024-04-25T09:26:29.190 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.198 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:407cca9e:::benchmark_data_smithi102_461693_object228:head# 2024-04-25T09:26:29.199 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.207 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:411befe3:::benchmark_data_smithi102_461693_object25:head# 2024-04-25T09:26:29.207 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.215 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:412d23d7:::benchmark_data_smithi102_461693_object174:head# 2024-04-25T09:26:29.216 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.223 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:426f5982:::benchmark_data_smithi102_461693_object7:head# 2024-04-25T09:26:29.224 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.232 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:443e3636:::benchmark_data_smithi102_461693_object201:head# 2024-04-25T09:26:29.232 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.240 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:45dbb99e:::benchmark_data_smithi102_461693_object139:head# 2024-04-25T09:26:29.240 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.249 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:462c230e:::benchmark_data_smithi102_461693_object189:head# 2024-04-25T09:26:29.249 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.257 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:46e420c2:::benchmark_data_smithi102_461693_object315:head# 2024-04-25T09:26:29.257 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.265 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:46ed67bb:::benchmark_data_smithi102_461693_object80:head# 2024-04-25T09:26:29.265 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.274 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:47d29e85:::benchmark_data_smithi102_461693_object89:head# 2024-04-25T09:26:29.274 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.282 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:485eaaa8:::benchmark_data_smithi102_461693_object66:head# 2024-04-25T09:26:29.282 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.290 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:49105468:::benchmark_data_smithi102_461693_object76:head# 2024-04-25T09:26:29.290 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.299 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:49d9b3e4:::benchmark_data_smithi102_461693_object268:head# 2024-04-25T09:26:29.299 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.307 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4a341421:::benchmark_data_smithi102_461693_object288:head# 2024-04-25T09:26:29.307 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.315 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4b1ffa35:::benchmark_data_smithi102_461693_object286:head# 2024-04-25T09:26:29.316 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.327 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4d5b2900:::benchmark_data_smithi102_461693_object23:head# 2024-04-25T09:26:29.327 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.332 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4d6d7ce3:::benchmark_data_smithi102_461693_object122:head# 2024-04-25T09:26:29.332 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.340 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4e5a5bae:::benchmark_data_smithi102_461693_object214:head# 2024-04-25T09:26:29.341 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.349 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4ee4fe84:::benchmark_data_smithi102_461693_object63:head# 2024-04-25T09:26:29.349 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.358 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4f4aa74e:::benchmark_data_smithi102_461693_object146:head# 2024-04-25T09:26:29.358 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.366 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4f7e5631:::benchmark_data_smithi102_461693_object251:head# 2024-04-25T09:26:29.366 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.374 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4f9e0719:::benchmark_data_smithi102_461693_object222:head# 2024-04-25T09:26:29.374 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.382 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:52c86389:::benchmark_data_smithi102_461693_object223:head# 2024-04-25T09:26:29.382 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.391 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5308d886:::benchmark_data_smithi102_461693_object45:head# 2024-04-25T09:26:29.391 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.399 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5403a8e1:::benchmark_data_smithi102_461693_object276:head# 2024-04-25T09:26:29.399 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.407 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:547cc0c2:::benchmark_data_smithi102_461693_object169:head# 2024-04-25T09:26:29.408 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.416 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:54f5df8b:::benchmark_data_smithi102_461693_object98:head# 2024-04-25T09:26:29.416 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.424 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5503c0c7:::benchmark_data_smithi102_461693_object133:head# 2024-04-25T09:26:29.424 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.432 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:557ea668:::benchmark_data_smithi102_461693_object65:head# 2024-04-25T09:26:29.432 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.441 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:57f2a07b:::benchmark_data_smithi102_461693_object293:head# 2024-04-25T09:26:29.441 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.449 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:580d47d3:::benchmark_data_smithi102_461693_object10:head# 2024-04-25T09:26:29.449 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.457 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:58a91fd0:::benchmark_data_smithi102_461693_object292:head# 2024-04-25T09:26:29.457 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.466 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5a9a4702:::benchmark_data_smithi102_461693_object30:head# 2024-04-25T09:26:29.466 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.474 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5b7a7287:::benchmark_data_smithi102_461693_object178:head# 2024-04-25T09:26:29.474 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.482 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5c075ec8:::benchmark_data_smithi102_461693_object91:head# 2024-04-25T09:26:29.482 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.491 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5d5f48ea:::benchmark_data_smithi102_461693_object302:head# 2024-04-25T09:26:29.491 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.499 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5d8f2267:::benchmark_data_smithi102_461693_object247:head# 2024-04-25T09:26:29.499 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.507 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5ec1726b:::benchmark_data_smithi102_461693_object126:head# 2024-04-25T09:26:29.508 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.516 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5fe7afcc:::benchmark_data_smithi102_461693_object238:head# 2024-04-25T09:26:29.516 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.524 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:60ff7266:::benchmark_data_smithi102_461693_object42:head# 2024-04-25T09:26:29.525 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.566 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:61051e5f:::benchmark_data_smithi102_461693_object128:head# 2024-04-25T09:26:29.567 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.575 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:638903a9:::benchmark_data_smithi102_461693_object271:head# 2024-04-25T09:26:29.575 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.583 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:64392431:::benchmark_data_smithi102_461693_object316:head# 2024-04-25T09:26:29.583 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.591 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6452bd93:::benchmark_data_smithi102_461693_object78:head# 2024-04-25T09:26:29.591 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.599 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6469f64c:::benchmark_data_smithi102_461693_object246:head# 2024-04-25T09:26:29.599 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.608 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:64ce6296:::benchmark_data_smithi102_461693_object100:head# 2024-04-25T09:26:29.608 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.673 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6590eadb:::benchmark_data_smithi102_461693_object159:head# 2024-04-25T09:26:29.673 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.716 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:665d634c:::benchmark_data_smithi102_461693_object90:head# 2024-04-25T09:26:29.716 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.725 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6748bc13:::benchmark_data_smithi102_461693_object36:head# 2024-04-25T09:26:29.725 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.734 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6785c648:::benchmark_data_smithi102_461693_object1:head# 2024-04-25T09:26:29.734 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.741 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6823a015:::benchmark_data_smithi102_461693_object254:head# 2024-04-25T09:26:29.742 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.750 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68498f01:::benchmark_data_smithi102_461693_object280:head# 2024-04-25T09:26:29.750 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.758 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68811066:::benchmark_data_smithi102_461693_object163:head# 2024-04-25T09:26:29.758 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.766 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68b09bf8:::benchmark_data_smithi102_461693_object224:head# 2024-04-25T09:26:29.766 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.775 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68c982f8:::benchmark_data_smithi102_461693_object97:head# 2024-04-25T09:26:29.775 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.783 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:69b24405:::benchmark_data_smithi102_461693_object294:head# 2024-04-25T09:26:29.783 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.791 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:69e50a6f:::benchmark_data_smithi102_461693_object143:head# 2024-04-25T09:26:29.791 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.800 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6a328a99:::benchmark_data_smithi102_461693_object120:head# 2024-04-25T09:26:29.800 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.808 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6aea4f95:::benchmark_data_smithi102_461693_object71:head# 2024-04-25T09:26:29.808 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.816 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6b81e8fb:::benchmark_data_smithi102_461693_object140:head# 2024-04-25T09:26:29.816 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.825 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6ba5d254:::benchmark_data_smithi102_461693_object68:head# 2024-04-25T09:26:29.825 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.833 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6bf4899e:::benchmark_data_smithi102_461693_object116:head# 2024-04-25T09:26:29.833 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.842 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6d1b960b:::benchmark_data_smithi102_461693_object35:head# 2024-04-25T09:26:29.842 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.850 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6df62e7a:::benchmark_data_smithi102_461693_object56:head# 2024-04-25T09:26:29.850 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.858 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6fd57c98:::benchmark_data_smithi102_461693_object162:head# 2024-04-25T09:26:29.858 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.867 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:70d697fa:::benchmark_data_smithi102_461693_object297:head# 2024-04-25T09:26:29.867 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.875 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:72074391:::benchmark_data_smithi102_461693_object102:head# 2024-04-25T09:26:29.875 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.883 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:74228ab3:::benchmark_data_smithi102_461693_object313:head# 2024-04-25T09:26:29.883 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.892 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:745a30ad:::benchmark_data_smithi102_461693_object75:head# 2024-04-25T09:26:29.892 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.900 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:762df197:::benchmark_data_smithi102_461693_object241:head# 2024-04-25T09:26:29.900 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.908 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7744fca5:::benchmark_data_smithi102_461693_object318:head# 2024-04-25T09:26:29.908 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.917 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:77ffafc4:::benchmark_data_smithi102_461693_object24:head# 2024-04-25T09:26:29.917 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.925 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:78013faf:::benchmark_data_smithi102_461693_object175:head# 2024-04-25T09:26:29.925 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.933 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7902734b:::benchmark_data_smithi102_461693_object135:head# 2024-04-25T09:26:29.933 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.942 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7958dd60:::benchmark_data_smithi102_461693_object190:head# 2024-04-25T09:26:29.942 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.950 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7a196166:::benchmark_data_smithi102_461693_object309:head# 2024-04-25T09:26:29.950 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.959 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7a932c37:::benchmark_data_smithi102_461693_object179:head# 2024-04-25T09:26:29.959 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.967 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7af1be98:::benchmark_data_smithi102_461693_object298:head# 2024-04-25T09:26:29.967 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.975 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7b48691d:::benchmark_data_smithi102_461693_object79:head# 2024-04-25T09:26:29.975 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.983 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7cc3111f:::benchmark_data_smithi102_461693_object44:head# 2024-04-25T09:26:29.984 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:29.992 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7cf5ad47:::benchmark_data_smithi102_461693_object121:head# 2024-04-25T09:26:29.992 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:30.000 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7d1cce5a:::benchmark_data_smithi102_461693_object206:head# 2024-04-25T09:26:30.000 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:30.009 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7dca102f:::benchmark_data_smithi102_461693_object202:head# 2024-04-25T09:26:30.009 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:30.017 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7df4b946:::benchmark_data_smithi102_461693_object245:head# 2024-04-25T09:26:30.017 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:30.025 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7e06601e:::benchmark_data_smithi102_461693_object252:head# 2024-04-25T09:26:30.025 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:30.034 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7f86f02e:::benchmark_data_smithi102_461693_object138:head# 2024-04-25T09:26:30.034 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:30.056 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7fb155ff:::benchmark_data_smithi102_461693_object188:head# 2024-04-25T09:26:30.056 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:30.115 INFO:tasks.workunit.client.0.smithi102.stdout:write_pg epoch 10 info 1.0( v 10'176 (0'0,10'176] local-lis/les=7/8 n=176 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:26:30.182 INFO:tasks.workunit.client.0.smithi102.stdout:Import successful 2024-04-25T09:26:30.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:151: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:26:31.404 INFO:tasks.workunit.client.0.smithi102.stdout: marking collection for removal 2024-04-25T09:26:31.405 INFO:tasks.workunit.client.0.smithi102.stdout:setting '_remove' omap key 2024-04-25T09:26:31.405 INFO:tasks.workunit.client.0.smithi102.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:26:31.534 INFO:tasks.workunit.client.0.smithi102.stdout:Remove successful 2024-04-25T09:26:31.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:152: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-04-25T09:26:32.580 INFO:tasks.workunit.client.0.smithi102.stdout: marking collection for removal 2024-04-25T09:26:32.581 INFO:tasks.workunit.client.0.smithi102.stdout:setting '_remove' omap key 2024-04-25T09:26:32.581 INFO:tasks.workunit.client.0.smithi102.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:26:32.693 INFO:tasks.workunit.client.0.smithi102.stdout:Remove successful 2024-04-25T09:26:33.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:154: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0.later --force 2024-04-25T09:26:33.817 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e229 pg_nums {1={20=1}} deleted_pools ) 2024-04-25T09:26:33.817 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.0 2024-04-25T09:26:33.817 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:000b3b92:::benchmark_data_smithi102_461693_object262:head# 2024-04-25T09:26:33.817 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:33.868 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:00374c2e:::benchmark_data_smithi102_461693_object216:head# 2024-04-25T09:26:33.869 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:33.893 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:01237c3c:::benchmark_data_smithi102_467209_object147:head# 2024-04-25T09:26:33.893 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:33.918 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0233f50b:::benchmark_data_smithi102_467209_object54:head# 2024-04-25T09:26:33.918 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:33.943 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0301c80c:::benchmark_data_smithi102_461693_object11:head# 2024-04-25T09:26:33.944 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:33.960 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:036651e6:::benchmark_data_smithi102_461693_object111:head# 2024-04-25T09:26:33.960 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:33.977 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0402916e:::benchmark_data_smithi102_461693_object183:head# 2024-04-25T09:26:33.977 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:33.993 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:046b3940:::benchmark_data_smithi102_461693_object104:head# 2024-04-25T09:26:33.994 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.010 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:046d0875:::benchmark_data_smithi102_467209_object139:head# 2024-04-25T09:26:34.010 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.027 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0540312f:::benchmark_data_smithi102_461693_object69:head# 2024-04-25T09:26:34.027 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.043 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:054ee2e1:::benchmark_data_smithi102_461693_object300:head# 2024-04-25T09:26:34.044 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.060 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:062f7c5e:::benchmark_data_smithi102_467209_object32:head# 2024-04-25T09:26:34.060 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.077 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:069b73e0:::benchmark_data_smithi102_461693_object195:head# 2024-04-25T09:26:34.077 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.093 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:06a51e53:::benchmark_data_smithi102_467209_object137:head# 2024-04-25T09:26:34.093 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.110 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:06bf8ef0:::benchmark_data_smithi102_467209_object156:head# 2024-04-25T09:26:34.110 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.151 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:07526ebf:::benchmark_data_smithi102_461693_object248:head# 2024-04-25T09:26:34.151 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.193 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0825096c:::benchmark_data_smithi102_467209_object15:head# 2024-04-25T09:26:34.193 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.244 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:084f14dd:::benchmark_data_smithi102_467209_object77:head# 2024-04-25T09:26:34.244 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.285 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:08aeff1a:::benchmark_data_smithi102_461693_object43:head# 2024-04-25T09:26:34.285 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.302 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:09569df1:::benchmark_data_smithi102_461693_object305:head# 2024-04-25T09:26:34.302 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.319 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:099020c4:::benchmark_data_smithi102_461693_object152:head# 2024-04-25T09:26:34.319 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.335 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:09a7c99f:::benchmark_data_smithi102_467209_object1:head# 2024-04-25T09:26:34.335 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.352 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0a47cb3d:::benchmark_data_smithi102_461693_object266:head# 2024-04-25T09:26:34.352 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.369 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0a7de0f7:::benchmark_data_smithi102_461693_object208:head# 2024-04-25T09:26:34.369 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.385 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0b43c519:::benchmark_data_smithi102_467209_object42:head# 2024-04-25T09:26:34.385 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.402 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0b6a70be:::benchmark_data_smithi102_461693_object85:head# 2024-04-25T09:26:34.402 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.419 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0c02d064:::benchmark_data_smithi102_467209_object112:head# 2024-04-25T09:26:34.419 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.435 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0cc78a03:::benchmark_data_smithi102_461693_object21:head# 2024-04-25T09:26:34.435 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.452 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0cd9cd5d:::benchmark_data_smithi102_461693_object160:head# 2024-04-25T09:26:34.452 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.469 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0cfde1d2:::benchmark_data_smithi102_467209_object105:head# 2024-04-25T09:26:34.469 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.485 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0deb35d3:::benchmark_data_smithi102_461693_object217:head# 2024-04-25T09:26:34.485 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.502 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0e8b2dd0:::benchmark_data_smithi102_461693_object148:head# 2024-04-25T09:26:34.502 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.519 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0f24b073:::benchmark_data_smithi102_461693_object317:head# 2024-04-25T09:26:34.519 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.535 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0ff8e14a:::benchmark_data_smithi102_467209_object3:head# 2024-04-25T09:26:34.536 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.552 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:10af48fb:::benchmark_data_smithi102_461693_object240:head# 2024-04-25T09:26:34.552 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.569 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:11692bdb:::benchmark_data_smithi102_461693_object27:head# 2024-04-25T09:26:34.569 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.586 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:128e9745:::benchmark_data_smithi102_461693_object314:head# 2024-04-25T09:26:34.586 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.602 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:12d5a862:::benchmark_data_smithi102_467209_object107:head# 2024-04-25T09:26:34.602 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.620 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:132fbbae:::benchmark_data_smithi102_461693_object92:head# 2024-04-25T09:26:34.620 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.636 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:14a6a0cd:::benchmark_data_smithi102_461693_object88:head# 2024-04-25T09:26:34.636 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.653 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:14f57f39:::benchmark_data_smithi102_467209_object58:head# 2024-04-25T09:26:34.653 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.669 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1512b06b:::benchmark_data_smithi102_461693_object272:head# 2024-04-25T09:26:34.669 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.686 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1629e659:::benchmark_data_smithi102_461693_object256:head# 2024-04-25T09:26:34.686 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.702 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:169aee60:::benchmark_data_smithi102_467209_object88:head# 2024-04-25T09:26:34.702 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.719 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:16f7bd8c:::benchmark_data_smithi102_461693_object64:head# 2024-04-25T09:26:34.719 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.736 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:17105e23:::benchmark_data_smithi102_467209_object136:head# 2024-04-25T09:26:34.736 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.752 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1786312c:::benchmark_data_smithi102_461693_object37:head# 2024-04-25T09:26:34.752 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.769 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:18b9710d:::benchmark_data_smithi102_467209_object131:head# 2024-04-25T09:26:34.769 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.786 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:19fa802a:::benchmark_data_smithi102_461693_object177:head# 2024-04-25T09:26:34.786 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.802 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1b20ea55:::benchmark_data_smithi102_467209_object2:head# 2024-04-25T09:26:34.803 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.819 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1b6923a0:::benchmark_data_smithi102_467209_object7:head# 2024-04-25T09:26:34.819 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.836 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1ba38020:::benchmark_data_smithi102_467209_object55:head# 2024-04-25T09:26:34.836 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.853 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1c114184:::benchmark_data_smithi102_467209_object8:head# 2024-04-25T09:26:34.853 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.869 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1c85f9b4:::benchmark_data_smithi102_467209_object25:head# 2024-04-25T09:26:34.869 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.886 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1d2fbc72:::benchmark_data_smithi102_461693_object253:head# 2024-04-25T09:26:34.886 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.903 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1dbe156c:::benchmark_data_smithi102_461693_object301:head# 2024-04-25T09:26:34.903 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.919 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1e43a575:::benchmark_data_smithi102_461693_object109:head# 2024-04-25T09:26:34.919 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.936 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1e98b968:::benchmark_data_smithi102_461693_object8:head# 2024-04-25T09:26:34.936 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.953 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1e9976c2:::benchmark_data_smithi102_461693_object31:head# 2024-04-25T09:26:34.953 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.969 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:202535ea:::benchmark_data_smithi102_467209_object121:head# 2024-04-25T09:26:34.969 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:34.986 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:20ac13d7:::benchmark_data_smithi102_461693_object319:head# 2024-04-25T09:26:34.986 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.003 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:217c431d:::benchmark_data_smithi102_467209_object84:head# 2024-04-25T09:26:35.003 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.019 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2279d77a:::benchmark_data_smithi102_461693_object157:head# 2024-04-25T09:26:35.019 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.036 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:22b40e23:::benchmark_data_smithi102_461693_object233:head# 2024-04-25T09:26:35.036 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.053 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:23111692:::benchmark_data_smithi102_461693_object14:head# 2024-04-25T09:26:35.053 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.103 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2331325a:::benchmark_data_smithi102_467209_object70:head# 2024-04-25T09:26:35.103 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.119 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:23426916:::benchmark_data_smithi102_461693_object164:head# 2024-04-25T09:26:35.119 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.137 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:238a3ae3:::benchmark_data_smithi102_461693_object211:head# 2024-04-25T09:26:35.137 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.153 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:239e27b3:::benchmark_data_smithi102_461693_object219:head# 2024-04-25T09:26:35.153 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.169 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:23b4b493:::benchmark_data_smithi102_461693_object192:head# 2024-04-25T09:26:35.170 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.186 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:24e58257:::benchmark_data_smithi102_461693_object141:head# 2024-04-25T09:26:35.186 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.203 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:24e6a109:::benchmark_data_smithi102_461693_object106:head# 2024-04-25T09:26:35.203 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.220 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:25506ed7:::benchmark_data_smithi102_467209_object100:head# 2024-04-25T09:26:35.220 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.236 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:25997359:::benchmark_data_smithi102_461693_object264:head# 2024-04-25T09:26:35.236 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.253 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:25b16926:::benchmark_data_smithi102_461693_object29:head# 2024-04-25T09:26:35.253 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.270 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:25e5150c:::benchmark_data_smithi102_467209_object126:head# 2024-04-25T09:26:35.270 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.286 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:26824afa:::benchmark_data_smithi102_461693_object94:head# 2024-04-25T09:26:35.287 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.303 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:26a55470:::benchmark_data_smithi102_461693_object274:head# 2024-04-25T09:26:35.303 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.320 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:272eb2c9:::benchmark_data_smithi102_461693_object149:head# 2024-04-25T09:26:35.320 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.336 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:273627fb:::benchmark_data_smithi102_461693_object125:head# 2024-04-25T09:26:35.336 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.353 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:273b3e12:::benchmark_data_smithi102_461693_object161:head# 2024-04-25T09:26:35.353 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.370 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:275d130a:::benchmark_data_smithi102_461693_object20:head# 2024-04-25T09:26:35.370 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.386 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:28694220:::benchmark_data_smithi102_461693_object136:head# 2024-04-25T09:26:35.386 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.403 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2a0bbe58:::benchmark_data_smithi102_461693_object199:head# 2024-04-25T09:26:35.403 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.420 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2a74d816:::benchmark_data_smithi102_467209_object99:head# 2024-04-25T09:26:35.420 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.436 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2a86184a:::benchmark_data_smithi102_467209_object39:head# 2024-04-25T09:26:35.436 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.453 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2afc2f48:::benchmark_data_smithi102_461693_object209:head# 2024-04-25T09:26:35.453 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.470 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2bbb9370:::benchmark_data_smithi102_461693_object287:head# 2024-04-25T09:26:35.470 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.487 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2bd520cd:::benchmark_data_smithi102_461693_object19:head# 2024-04-25T09:26:35.487 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.503 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2be202af:::benchmark_data_smithi102_461693_object234:head# 2024-04-25T09:26:35.503 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.528 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2c0b4651:::benchmark_data_smithi102_461693_object74:head# 2024-04-25T09:26:35.528 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.545 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2c9a727c:::benchmark_data_smithi102_461693_object165:head# 2024-04-25T09:26:35.545 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.562 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2cf2f038:::benchmark_data_smithi102_467209_object118:head# 2024-04-25T09:26:35.562 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.578 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2d3d0890:::benchmark_data_smithi102_461693_object50:head# 2024-04-25T09:26:35.578 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.595 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2d3fef0a:::benchmark_data_smithi102_461693_object173:head# 2024-04-25T09:26:35.595 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.612 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2d4d8992:::benchmark_data_smithi102_461693_object53:head# 2024-04-25T09:26:35.612 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.660 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2d554bd3:::benchmark_data_smithi102_467209_object65:head# 2024-04-25T09:26:35.660 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.695 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2daabe1c:::benchmark_data_smithi102_467209_object44:head# 2024-04-25T09:26:35.695 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.720 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2ee095c4:::benchmark_data_smithi102_461693_object58:head# 2024-04-25T09:26:35.721 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.737 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2fd07b30:::benchmark_data_smithi102_461693_object194:head# 2024-04-25T09:26:35.737 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.754 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2fee0198:::benchmark_data_smithi102_461693_object2:head# 2024-04-25T09:26:35.754 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.771 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:31f0e698:::benchmark_data_smithi102_467209_object6:head# 2024-04-25T09:26:35.771 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.787 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:34891980:::benchmark_data_smithi102_467209_object0:head# 2024-04-25T09:26:35.787 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.804 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:34e9f16c:::benchmark_data_smithi102_461693_object260:head# 2024-04-25T09:26:35.804 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.846 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:35004fae:::benchmark_data_smithi102_467209_object68:head# 2024-04-25T09:26:35.846 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.862 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:355453a0:::benchmark_data_smithi102_461693_object118:head# 2024-04-25T09:26:35.862 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.879 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:355b9f1f:::benchmark_data_smithi102_467209_object110:head# 2024-04-25T09:26:35.879 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.896 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:363cd543:::benchmark_data_smithi102_461693_object34:head# 2024-04-25T09:26:35.896 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.912 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:36551f3c:::benchmark_data_smithi102_461693_object51:head# 2024-04-25T09:26:35.913 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.929 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3693711a:::benchmark_data_smithi102_461693_object306:head# 2024-04-25T09:26:35.929 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.946 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:37225bcc:::benchmark_data_smithi102_461693_object117:head# 2024-04-25T09:26:35.946 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.962 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:37c0cca3:::benchmark_data_smithi102_461693_object282:head# 2024-04-25T09:26:35.962 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.979 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:389b2f68:::benchmark_data_smithi102_461693_object54:head# 2024-04-25T09:26:35.979 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:35.996 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:38c55045:::benchmark_data_smithi102_467209_object143:head# 2024-04-25T09:26:35.996 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.012 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:38f5ec38:::benchmark_data_smithi102_461693_object12:head# 2024-04-25T09:26:36.012 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.029 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:396efdd2:::benchmark_data_smithi102_467209_object63:head# 2024-04-25T09:26:36.029 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.054 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:39a2e424:::benchmark_data_smithi102_461693_object16:head# 2024-04-25T09:26:36.054 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.071 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3b46a605:::benchmark_data_smithi102_467209_object57:head# 2024-04-25T09:26:36.071 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.088 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3b72c6d5:::benchmark_data_smithi102_461693_object28:head# 2024-04-25T09:26:36.088 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.104 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3b87185e:::benchmark_data_smithi102_467209_object45:head# 2024-04-25T09:26:36.105 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.121 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3bca4278:::benchmark_data_smithi102_461693_object6:head# 2024-04-25T09:26:36.121 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.177 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3c246a75:::benchmark_data_smithi102_467209_object83:head# 2024-04-25T09:26:36.177 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.228 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3c78f5d8:::benchmark_data_smithi102_461693_object124:head# 2024-04-25T09:26:36.228 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.277 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3ccb0f2b:::benchmark_data_smithi102_461693_object144:head# 2024-04-25T09:26:36.278 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.304 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3d133806:::benchmark_data_smithi102_461693_object243:head# 2024-04-25T09:26:36.305 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.321 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3d2ad51e:::benchmark_data_smithi102_461693_object38:head# 2024-04-25T09:26:36.321 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.338 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3d611f0a:::benchmark_data_smithi102_461693_object32:head# 2024-04-25T09:26:36.338 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.354 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3f6d6ad9:::benchmark_data_smithi102_461693_object204:head# 2024-04-25T09:26:36.355 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.371 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3f7d2427:::benchmark_data_smithi102_461693_object205:head# 2024-04-25T09:26:36.371 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.413 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:40581110:::benchmark_data_smithi102_461693_object147:head# 2024-04-25T09:26:36.413 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.429 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4066b8f9:::benchmark_data_smithi102_461693_object103:head# 2024-04-25T09:26:36.430 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.446 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:407cca9e:::benchmark_data_smithi102_461693_object228:head# 2024-04-25T09:26:36.446 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.463 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:411219b0:::benchmark_data_smithi102_467209_object154:head# 2024-04-25T09:26:36.463 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.479 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:411befe3:::benchmark_data_smithi102_461693_object25:head# 2024-04-25T09:26:36.480 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.496 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:412d23d7:::benchmark_data_smithi102_461693_object174:head# 2024-04-25T09:26:36.496 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.513 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:426f5982:::benchmark_data_smithi102_461693_object7:head# 2024-04-25T09:26:36.513 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.529 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:44133081:::benchmark_data_smithi102_467209_object69:head# 2024-04-25T09:26:36.529 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.546 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:443e3636:::benchmark_data_smithi102_461693_object201:head# 2024-04-25T09:26:36.546 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.571 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:44a791f4:::benchmark_data_smithi102_467209_object146:head# 2024-04-25T09:26:36.571 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.588 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:45dbb99e:::benchmark_data_smithi102_461693_object139:head# 2024-04-25T09:26:36.588 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.604 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:462c230e:::benchmark_data_smithi102_461693_object189:head# 2024-04-25T09:26:36.605 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.621 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:46e420c2:::benchmark_data_smithi102_461693_object315:head# 2024-04-25T09:26:36.621 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.638 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:46ed67bb:::benchmark_data_smithi102_461693_object80:head# 2024-04-25T09:26:36.638 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.655 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:47d29e85:::benchmark_data_smithi102_461693_object89:head# 2024-04-25T09:26:36.655 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.671 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:485eaaa8:::benchmark_data_smithi102_461693_object66:head# 2024-04-25T09:26:36.671 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.688 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:49105468:::benchmark_data_smithi102_461693_object76:head# 2024-04-25T09:26:36.688 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.705 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:49d9b3e4:::benchmark_data_smithi102_461693_object268:head# 2024-04-25T09:26:36.705 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.721 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4a341421:::benchmark_data_smithi102_461693_object288:head# 2024-04-25T09:26:36.721 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.738 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4a657f62:::benchmark_data_smithi102_467209_object132:head# 2024-04-25T09:26:36.738 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.755 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4b1ffa35:::benchmark_data_smithi102_461693_object286:head# 2024-04-25T09:26:36.755 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.771 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4cab95a8:::benchmark_data_smithi102_467209_object51:head# 2024-04-25T09:26:36.771 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.788 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4cfe58df:::benchmark_data_smithi102_467209_object122:head# 2024-04-25T09:26:36.788 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.805 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4d5b2900:::benchmark_data_smithi102_461693_object23:head# 2024-04-25T09:26:36.805 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.821 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4d6d7ce3:::benchmark_data_smithi102_461693_object122:head# 2024-04-25T09:26:36.822 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.838 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4e5a5bae:::benchmark_data_smithi102_461693_object214:head# 2024-04-25T09:26:36.838 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.855 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4ee4fe84:::benchmark_data_smithi102_461693_object63:head# 2024-04-25T09:26:36.855 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.871 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4f022233:::benchmark_data_smithi102_467209_object81:head# 2024-04-25T09:26:36.872 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.888 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4f4aa74e:::benchmark_data_smithi102_461693_object146:head# 2024-04-25T09:26:36.888 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.930 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4f7e5631:::benchmark_data_smithi102_461693_object251:head# 2024-04-25T09:26:36.930 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.947 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4f9e0719:::benchmark_data_smithi102_461693_object222:head# 2024-04-25T09:26:36.947 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.963 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:507c4ede:::benchmark_data_smithi102_467209_object11:head# 2024-04-25T09:26:36.964 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.980 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:51ef0d4b:::benchmark_data_smithi102_467209_object75:head# 2024-04-25T09:26:36.980 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:36.997 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:52c86389:::benchmark_data_smithi102_461693_object223:head# 2024-04-25T09:26:36.997 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.013 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5308d886:::benchmark_data_smithi102_461693_object45:head# 2024-04-25T09:26:37.013 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.030 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5403a8e1:::benchmark_data_smithi102_461693_object276:head# 2024-04-25T09:26:37.030 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.047 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:54221da9:::benchmark_data_smithi102_467209_object59:head# 2024-04-25T09:26:37.047 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.063 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:547cc0c2:::benchmark_data_smithi102_461693_object169:head# 2024-04-25T09:26:37.063 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.080 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:54f5df8b:::benchmark_data_smithi102_461693_object98:head# 2024-04-25T09:26:37.080 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.097 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5503c0c7:::benchmark_data_smithi102_461693_object133:head# 2024-04-25T09:26:37.097 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.114 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:557ea668:::benchmark_data_smithi102_461693_object65:head# 2024-04-25T09:26:37.114 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.130 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:56c5df50:::benchmark_data_smithi102_467209_object61:head# 2024-04-25T09:26:37.130 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.147 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:57f2a07b:::benchmark_data_smithi102_461693_object293:head# 2024-04-25T09:26:37.147 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.163 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:580d47d3:::benchmark_data_smithi102_461693_object10:head# 2024-04-25T09:26:37.164 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.180 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:58a91fd0:::benchmark_data_smithi102_461693_object292:head# 2024-04-25T09:26:37.180 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.197 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5a9a4702:::benchmark_data_smithi102_461693_object30:head# 2024-04-25T09:26:37.197 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.214 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5b1e4f29:::benchmark_data_smithi102_467209_object76:head# 2024-04-25T09:26:37.214 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.230 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5b7a7287:::benchmark_data_smithi102_461693_object178:head# 2024-04-25T09:26:37.230 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.247 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5c075ec8:::benchmark_data_smithi102_461693_object91:head# 2024-04-25T09:26:37.247 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.264 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5ce181a5:::benchmark_data_smithi102_467209_object134:head# 2024-04-25T09:26:37.264 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.280 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5d2a150c:::benchmark_data_smithi102_467209_object125:head# 2024-04-25T09:26:37.281 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.297 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5d5f48ea:::benchmark_data_smithi102_461693_object302:head# 2024-04-25T09:26:37.297 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.314 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5d8f2267:::benchmark_data_smithi102_461693_object247:head# 2024-04-25T09:26:37.314 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.330 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5e9e18aa:::benchmark_data_smithi102_467209_object123:head# 2024-04-25T09:26:37.330 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.347 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5ec1726b:::benchmark_data_smithi102_461693_object126:head# 2024-04-25T09:26:37.347 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.364 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5ecc9427:::benchmark_data_smithi102_467209_object127:head# 2024-04-25T09:26:37.364 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.380 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5fc8778c:::benchmark_data_smithi102_467209_object155:head# 2024-04-25T09:26:37.381 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.397 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5fe7afcc:::benchmark_data_smithi102_461693_object238:head# 2024-04-25T09:26:37.397 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.414 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6009192c:::benchmark_data_smithi102_467209_object157:head# 2024-04-25T09:26:37.414 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.430 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:60ff7266:::benchmark_data_smithi102_461693_object42:head# 2024-04-25T09:26:37.431 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.447 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:61051e5f:::benchmark_data_smithi102_461693_object128:head# 2024-04-25T09:26:37.447 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.464 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:638903a9:::benchmark_data_smithi102_461693_object271:head# 2024-04-25T09:26:37.464 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.481 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:64392431:::benchmark_data_smithi102_461693_object316:head# 2024-04-25T09:26:37.481 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.497 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6452bd93:::benchmark_data_smithi102_461693_object78:head# 2024-04-25T09:26:37.497 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.539 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6469f64c:::benchmark_data_smithi102_461693_object246:head# 2024-04-25T09:26:37.539 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.556 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:64ce6296:::benchmark_data_smithi102_461693_object100:head# 2024-04-25T09:26:37.556 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.573 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6590eadb:::benchmark_data_smithi102_461693_object159:head# 2024-04-25T09:26:37.573 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.589 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:665d634c:::benchmark_data_smithi102_461693_object90:head# 2024-04-25T09:26:37.589 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.606 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6748bc13:::benchmark_data_smithi102_461693_object36:head# 2024-04-25T09:26:37.606 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.648 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6785c648:::benchmark_data_smithi102_461693_object1:head# 2024-04-25T09:26:37.648 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.689 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6823a015:::benchmark_data_smithi102_461693_object254:head# 2024-04-25T09:26:37.689 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.706 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68498f01:::benchmark_data_smithi102_461693_object280:head# 2024-04-25T09:26:37.706 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.722 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68811066:::benchmark_data_smithi102_461693_object163:head# 2024-04-25T09:26:37.722 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.739 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68b09bf8:::benchmark_data_smithi102_461693_object224:head# 2024-04-25T09:26:37.739 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.756 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68c982f8:::benchmark_data_smithi102_461693_object97:head# 2024-04-25T09:26:37.756 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.772 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:69b24405:::benchmark_data_smithi102_461693_object294:head# 2024-04-25T09:26:37.773 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.789 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:69e50a6f:::benchmark_data_smithi102_461693_object143:head# 2024-04-25T09:26:37.789 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.806 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6a328a99:::benchmark_data_smithi102_461693_object120:head# 2024-04-25T09:26:37.806 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.823 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6aea4f95:::benchmark_data_smithi102_461693_object71:head# 2024-04-25T09:26:37.823 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.839 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6b81e8fb:::benchmark_data_smithi102_461693_object140:head# 2024-04-25T09:26:37.839 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.856 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6ba5d254:::benchmark_data_smithi102_461693_object68:head# 2024-04-25T09:26:37.856 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.873 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6bb50989:::benchmark_data_smithi102_467209_object9:head# 2024-04-25T09:26:37.873 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.889 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6bf4899e:::benchmark_data_smithi102_461693_object116:head# 2024-04-25T09:26:37.889 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.906 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6c3afad8:::benchmark_data_smithi102_467209_object50:head# 2024-04-25T09:26:37.906 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.923 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6d1b960b:::benchmark_data_smithi102_461693_object35:head# 2024-04-25T09:26:37.923 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.939 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6df62e7a:::benchmark_data_smithi102_461693_object56:head# 2024-04-25T09:26:37.940 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.956 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6e487e28:::benchmark_data_smithi102_467209_object129:head# 2024-04-25T09:26:37.956 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.973 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6f5152dc:::benchmark_data_smithi102_467209_object14:head# 2024-04-25T09:26:37.973 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:37.989 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6fd57c98:::benchmark_data_smithi102_461693_object162:head# 2024-04-25T09:26:37.990 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.006 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:70b3821b:::benchmark_data_smithi102_467209_object60:head# 2024-04-25T09:26:38.007 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.023 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:70d697fa:::benchmark_data_smithi102_461693_object297:head# 2024-04-25T09:26:38.023 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.040 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:71abbb27:::benchmark_data_smithi102_467209_object19:head# 2024-04-25T09:26:38.040 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.056 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:72074391:::benchmark_data_smithi102_461693_object102:head# 2024-04-25T09:26:38.056 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.073 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:74228ab3:::benchmark_data_smithi102_461693_object313:head# 2024-04-25T09:26:38.073 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.090 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:745a30ad:::benchmark_data_smithi102_461693_object75:head# 2024-04-25T09:26:38.090 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.106 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:75aa73ef:::benchmark_data_smithi102_467209_object20:head# 2024-04-25T09:26:38.106 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.136 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:762df197:::benchmark_data_smithi102_461693_object241:head# 2024-04-25T09:26:38.136 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.186 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7744fca5:::benchmark_data_smithi102_461693_object318:head# 2024-04-25T09:26:38.186 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.236 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:779d6c73:::benchmark_data_smithi102_467209_object153:head# 2024-04-25T09:26:38.236 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.282 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:77ae3015:::benchmark_data_smithi102_467209_object78:head# 2024-04-25T09:26:38.282 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.298 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:77dae1b7:::benchmark_data_smithi102_467209_object30:head# 2024-04-25T09:26:38.298 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.315 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:77ffafc4:::benchmark_data_smithi102_461693_object24:head# 2024-04-25T09:26:38.315 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.331 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:78013faf:::benchmark_data_smithi102_461693_object175:head# 2024-04-25T09:26:38.332 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.348 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7902734b:::benchmark_data_smithi102_461693_object135:head# 2024-04-25T09:26:38.348 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7958dd60:::benchmark_data_smithi102_461693_object190:head# 2024-04-25T09:26:38.365 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.381 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7a196166:::benchmark_data_smithi102_461693_object309:head# 2024-04-25T09:26:38.381 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.398 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7a729e51:::benchmark_data_smithi102_467209_object130:head# 2024-04-25T09:26:38.398 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.415 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7a932c37:::benchmark_data_smithi102_461693_object179:head# 2024-04-25T09:26:38.415 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.432 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7aea4ed6:::benchmark_data_smithi102_467209_object37:head# 2024-04-25T09:26:38.432 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.448 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7af1be98:::benchmark_data_smithi102_461693_object298:head# 2024-04-25T09:26:38.449 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.465 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7b48691d:::benchmark_data_smithi102_461693_object79:head# 2024-04-25T09:26:38.465 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.482 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7c3929ca:::benchmark_data_smithi102_467209_object85:head# 2024-04-25T09:26:38.482 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.498 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7cc3111f:::benchmark_data_smithi102_461693_object44:head# 2024-04-25T09:26:38.498 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.515 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7cf5ad47:::benchmark_data_smithi102_461693_object121:head# 2024-04-25T09:26:38.515 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.532 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7d1cce5a:::benchmark_data_smithi102_461693_object206:head# 2024-04-25T09:26:38.532 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.548 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7dca102f:::benchmark_data_smithi102_461693_object202:head# 2024-04-25T09:26:38.548 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.565 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7df4b946:::benchmark_data_smithi102_461693_object245:head# 2024-04-25T09:26:38.565 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.582 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7e06601e:::benchmark_data_smithi102_461693_object252:head# 2024-04-25T09:26:38.582 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.598 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7e8819ca:::benchmark_data_smithi102_467209_object92:head# 2024-04-25T09:26:38.599 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.615 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7f86f02e:::benchmark_data_smithi102_461693_object138:head# 2024-04-25T09:26:38.615 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.632 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7fb155ff:::benchmark_data_smithi102_461693_object188:head# 2024-04-25T09:26:38.632 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.648 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:804ec195:::benchmark_data_smithi102_467209_object16:head# 2024-04-25T09:26:38.649 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.665 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:807e8dc6:::benchmark_data_smithi102_467209_object74:head# 2024-04-25T09:26:38.665 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.682 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:808c9235:::benchmark_data_smithi102_467209_object34:head# 2024-04-25T09:26:38.682 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.699 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:80decefa:::benchmark_data_smithi102_461693_object215:head# 2024-04-25T09:26:38.699 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.715 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:81279b41:::benchmark_data_smithi102_467209_object87:head# 2024-04-25T09:26:38.715 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.732 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:821259f4:::benchmark_data_smithi102_467209_object101:head# 2024-04-25T09:26:38.732 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.749 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:82443fae:::benchmark_data_smithi102_467209_object40:head# 2024-04-25T09:26:38.749 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.799 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:82504851:::benchmark_data_smithi102_461693_object59:head# 2024-04-25T09:26:38.799 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.816 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:82bc124c:::benchmark_data_smithi102_461693_object242:head# 2024-04-25T09:26:38.816 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.832 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:83944c13:::benchmark_data_smithi102_467209_object82:head# 2024-04-25T09:26:38.832 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.849 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:83a6ed54:::benchmark_data_smithi102_467209_object120:head# 2024-04-25T09:26:38.849 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.866 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:84376e46:::benchmark_data_smithi102_467209_object12:head# 2024-04-25T09:26:38.866 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.882 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:883cb801:::benchmark_data_smithi102_461693_object150:head# 2024-04-25T09:26:38.882 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.932 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:887d5142:::benchmark_data_smithi102_467209_object151:head# 2024-04-25T09:26:38.932 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.949 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:889e9aa5:::benchmark_data_smithi102_461693_object250:head# 2024-04-25T09:26:38.949 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.965 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:88bc3744:::benchmark_data_smithi102_467209_object79:head# 2024-04-25T09:26:38.966 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.982 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:890093a8:::benchmark_data_smithi102_467209_object109:head# 2024-04-25T09:26:38.982 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:38.999 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:890d9aaf:::benchmark_data_smithi102_461693_object96:head# 2024-04-25T09:26:38.999 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.015 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8a093910:::benchmark_data_smithi102_461693_object15:head# 2024-04-25T09:26:39.016 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.032 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8aedfdc0:::benchmark_data_smithi102_461693_object193:head# 2024-04-25T09:26:39.032 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.049 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8afeae38:::benchmark_data_smithi102_461693_object221:head# 2024-04-25T09:26:39.049 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.066 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8b078005:::benchmark_data_smithi102_461693_object60:head# 2024-04-25T09:26:39.066 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.082 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8b4ee6c4:::benchmark_data_smithi102_461693_object166:head# 2024-04-25T09:26:39.082 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.099 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8c12de8e:::benchmark_data_smithi102_467209_object21:head# 2024-04-25T09:26:39.099 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.116 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8ca6a195:::benchmark_data_smithi102_461693_object172:head# 2024-04-25T09:26:39.116 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.132 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8cdff5e0:::benchmark_data_smithi102_461693_object17:head# 2024-04-25T09:26:39.132 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.149 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8d3cebfe:::benchmark_data_smithi102_467209_object47:head# 2024-04-25T09:26:39.149 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.166 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8d571cff:::benchmark_data_smithi102_461693_object255:head# 2024-04-25T09:26:39.166 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.182 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8eeaf510:::benchmark_data_smithi102_461693_object22:head# 2024-04-25T09:26:39.182 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.199 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8f3dbf2e:::benchmark_data_smithi102_461693_object110:head# 2024-04-25T09:26:39.199 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.216 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9036035d:::benchmark_data_smithi102_467209_object94:head# 2024-04-25T09:26:39.216 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.232 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:910c4b69:::benchmark_data_smithi102_467209_object17:head# 2024-04-25T09:26:39.233 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.249 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:91d0aef9:::benchmark_data_smithi102_461693_object132:head# 2024-04-25T09:26:39.249 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.266 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9269286e:::benchmark_data_smithi102_467209_object26:head# 2024-04-25T09:26:39.266 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.282 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:92c15cd6:::benchmark_data_smithi102_467209_object22:head# 2024-04-25T09:26:39.283 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.299 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:93bbd15e:::benchmark_data_smithi102_461693_object311:head# 2024-04-25T09:26:39.299 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.316 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:93cb41d3:::benchmark_data_smithi102_467209_object135:head# 2024-04-25T09:26:39.316 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.332 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:941e50ae:::benchmark_data_smithi102_467209_object113:head# 2024-04-25T09:26:39.333 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.349 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:942bc514:::benchmark_data_smithi102_461693_object77:head# 2024-04-25T09:26:39.349 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.366 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:94d0ab29:::benchmark_data_smithi102_461693_object285:head# 2024-04-25T09:26:39.366 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.383 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:94ec3b32:::benchmark_data_smithi102_461693_object72:head# 2024-04-25T09:26:39.383 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.399 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:95092f81:::benchmark_data_smithi102_461693_object220:head# 2024-04-25T09:26:39.399 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.416 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:95a9ddc7:::benchmark_data_smithi102_467209_object142:head# 2024-04-25T09:26:39.416 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.433 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:95b06a69:::benchmark_data_smithi102_461693_object67:head# 2024-04-25T09:26:39.433 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.450 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:95f571da:::benchmark_data_smithi102_467209_object66:head# 2024-04-25T09:26:39.450 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.466 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:969b0eab:::benchmark_data_smithi102_467209_object106:head# 2024-04-25T09:26:39.466 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.483 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:96ade7c2:::benchmark_data_smithi102_461693_object115:head# 2024-04-25T09:26:39.483 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.500 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:96e9fce0:::benchmark_data_smithi102_467209_object38:head# 2024-04-25T09:26:39.500 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.516 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:971cae83:::benchmark_data_smithi102_461693_object196:head# 2024-04-25T09:26:39.516 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.533 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:978b03f9:::benchmark_data_smithi102_461693_object236:head# 2024-04-25T09:26:39.533 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.550 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9862f725:::benchmark_data_smithi102_467209_object133:head# 2024-04-25T09:26:39.550 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.566 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:98701baa:::benchmark_data_smithi102_461693_object55:head# 2024-04-25T09:26:39.566 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.583 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:98e4a738:::benchmark_data_smithi102_461693_object127:head# 2024-04-25T09:26:39.583 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.600 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9aa9891c:::benchmark_data_smithi102_461693_object5:head# 2024-04-25T09:26:39.600 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.626 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9afe2884:::benchmark_data_smithi102_461693_object273:head# 2024-04-25T09:26:39.626 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.676 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9bd5d2e8:::benchmark_data_smithi102_461693_object265:head# 2024-04-25T09:26:39.676 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.716 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9bf31ddf:::benchmark_data_smithi102_461693_object83:head# 2024-04-25T09:26:39.717 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.733 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9c9b760d:::benchmark_data_smithi102_461693_object291:head# 2024-04-25T09:26:39.733 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.750 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9c9c7e9b:::benchmark_data_smithi102_461693_object70:head# 2024-04-25T09:26:39.750 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.766 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9cc82558:::benchmark_data_smithi102_461693_object108:head# 2024-04-25T09:26:39.767 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.792 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9d35491a:::benchmark_data_smithi102_467209_object158:head# 2024-04-25T09:26:39.792 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.817 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9d570aaf:::benchmark_data_smithi102_467209_object13:head# 2024-04-25T09:26:39.817 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.833 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9e60200b:::benchmark_data_smithi102_461693_object26:head# 2024-04-25T09:26:39.833 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.850 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9ebdc7a5:::benchmark_data_smithi102_461693_object18:head# 2024-04-25T09:26:39.850 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.867 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9eccdf33:::benchmark_data_smithi102_461693_object131:head# 2024-04-25T09:26:39.867 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.883 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9f958dac:::benchmark_data_smithi102_467209_object48:head# 2024-04-25T09:26:39.884 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.900 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9fcd1944:::benchmark_data_smithi102_461693_object198:head# 2024-04-25T09:26:39.900 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.917 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9fcee8c9:::benchmark_data_smithi102_461693_object263:head# 2024-04-25T09:26:39.917 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.933 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a109394d:::benchmark_data_smithi102_461693_object49:head# 2024-04-25T09:26:39.934 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.950 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a284d2b1:::benchmark_data_smithi102_461693_object40:head# 2024-04-25T09:26:39.950 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.967 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a2c77c71:::benchmark_data_smithi102_467209_object72:head# 2024-04-25T09:26:39.967 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:39.992 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a2ff1fbf:::benchmark_data_smithi102_461693_object52:head# 2024-04-25T09:26:39.992 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.008 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a32db852:::benchmark_data_smithi102_461693_object232:head# 2024-04-25T09:26:40.009 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.025 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a34ad8b3:::benchmark_data_smithi102_467209_object56:head# 2024-04-25T09:26:40.025 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.042 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a4f4d3a4:::benchmark_data_smithi102_461693_object261:head# 2024-04-25T09:26:40.042 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.058 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a7538180:::benchmark_data_smithi102_461693_object308:head# 2024-04-25T09:26:40.058 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.075 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aa0c2ca6:::benchmark_data_smithi102_461693_object213:head# 2024-04-25T09:26:40.075 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aa2b501d:::benchmark_data_smithi102_461693_object218:head# 2024-04-25T09:26:40.092 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.108 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aa40adca:::benchmark_data_smithi102_467209_object24:head# 2024-04-25T09:26:40.108 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.145 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aae249da:::benchmark_data_smithi102_461693_object182:head# 2024-04-25T09:26:40.145 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.204 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:aaf7ac98:::benchmark_data_smithi102_467209_object49:head# 2024-04-25T09:26:40.204 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.254 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ab0fe793:::benchmark_data_smithi102_467209_object159:head# 2024-04-25T09:26:40.254 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.300 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ab44e8cd:::benchmark_data_smithi102_467209_object86:head# 2024-04-25T09:26:40.300 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.317 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:abbc718e:::benchmark_data_smithi102_467209_object29:head# 2024-04-25T09:26:40.317 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.334 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac080002:::benchmark_data_smithi102_461693_object13:head# 2024-04-25T09:26:40.334 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.350 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac459348:::benchmark_data_smithi102_467209_object124:head# 2024-04-25T09:26:40.350 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.367 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac64182c:::benchmark_data_smithi102_461693_object295:head# 2024-04-25T09:26:40.367 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.384 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac73d9db:::benchmark_data_smithi102_467209_object95:head# 2024-04-25T09:26:40.384 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.400 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac96a035:::benchmark_data_smithi102_461693_object46:head# 2024-04-25T09:26:40.401 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.417 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:acd4c3d0:::benchmark_data_smithi102_461693_object226:head# 2024-04-25T09:26:40.417 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.434 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ad92f4ab:::benchmark_data_smithi102_461693_object303:head# 2024-04-25T09:26:40.434 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.450 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:adef340d:::benchmark_data_smithi102_461693_object61:head# 2024-04-25T09:26:40.450 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.467 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:afa99dfb:::benchmark_data_smithi102_467209_object67:head# 2024-04-25T09:26:40.467 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.484 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b1d35c85:::benchmark_data_smithi102_461693_object200:head# 2024-04-25T09:26:40.484 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.500 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b1d73141:::benchmark_data_smithi102_461693_object112:head# 2024-04-25T09:26:40.500 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.517 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b2c8f166:::benchmark_data_smithi102_467209_object46:head# 2024-04-25T09:26:40.517 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.534 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b4852218:::benchmark_data_smithi102_461693_object277:head# 2024-04-25T09:26:40.534 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.551 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b56c2919:::benchmark_data_smithi102_461693_object270:head# 2024-04-25T09:26:40.551 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.567 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b56e0ab2:::benchmark_data_smithi102_461693_object231:head# 2024-04-25T09:26:40.567 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.584 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b5c35fa0:::benchmark_data_smithi102_461693_object130:head# 2024-04-25T09:26:40.584 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.601 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b5d52fee:::benchmark_data_smithi102_467209_object36:head# 2024-04-25T09:26:40.601 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.617 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b79610bd:::benchmark_data_smithi102_467209_object150:head# 2024-04-25T09:26:40.617 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.634 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b7e1813e:::benchmark_data_smithi102_461693_object185:head# 2024-04-25T09:26:40.634 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.650 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:26:40.651 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.667 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b8772720:::benchmark_data_smithi102_461693_object289:head# 2024-04-25T09:26:40.668 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.692 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b925424a:::benchmark_data_smithi102_467209_object114:head# 2024-04-25T09:26:40.693 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.709 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bba685cf:::benchmark_data_smithi102_461693_object155:head# 2024-04-25T09:26:40.709 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.726 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc0f76dd:::benchmark_data_smithi102_467209_object71:head# 2024-04-25T09:26:40.726 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.742 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc36f44b:::benchmark_data_smithi102_461693_object296:head# 2024-04-25T09:26:40.743 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.759 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc9448f0:::benchmark_data_smithi102_461693_object82:head# 2024-04-25T09:26:40.760 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.776 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bd20be31:::benchmark_data_smithi102_461693_object93:head# 2024-04-25T09:26:40.776 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.792 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bda0949f:::benchmark_data_smithi102_461693_object181:head# 2024-04-25T09:26:40.792 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.809 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bdd24e14:::benchmark_data_smithi102_467209_object73:head# 2024-04-25T09:26:40.809 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.826 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bde44e2f:::benchmark_data_smithi102_461693_object95:head# 2024-04-25T09:26:40.826 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.842 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:be6d30be:::benchmark_data_smithi102_461693_object207:head# 2024-04-25T09:26:40.842 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.859 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bf7281fd:::benchmark_data_smithi102_467209_object145:head# 2024-04-25T09:26:40.859 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.876 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bf7d046b:::benchmark_data_smithi102_461693_object278:head# 2024-04-25T09:26:40.876 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.892 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c1eaa85c:::benchmark_data_smithi102_461693_object137:head# 2024-04-25T09:26:40.893 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.909 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c2ae1d76:::benchmark_data_smithi102_467209_object141:head# 2024-04-25T09:26:40.909 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.926 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c2d6c8cd:::benchmark_data_smithi102_467209_object91:head# 2024-04-25T09:26:40.926 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.943 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c37f1115:::benchmark_data_smithi102_461693_object310:head# 2024-04-25T09:26:40.943 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.959 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c402eb86:::benchmark_data_smithi102_467209_object149:head# 2024-04-25T09:26:40.959 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.976 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c50db599:::benchmark_data_smithi102_461693_object176:head# 2024-04-25T09:26:40.976 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:40.993 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c53ef393:::benchmark_data_smithi102_461693_object279:head# 2024-04-25T09:26:40.993 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.009 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c53f44d8:::benchmark_data_smithi102_461693_object239:head# 2024-04-25T09:26:41.009 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.026 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c551c4c0:::benchmark_data_smithi102_461693_object123:head# 2024-04-25T09:26:41.026 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.043 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c55d9b9d:::benchmark_data_smithi102_461693_object237:head# 2024-04-25T09:26:41.043 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.059 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c5e9ecd2:::benchmark_data_smithi102_467209_object5:head# 2024-04-25T09:26:41.059 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.076 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c6ad85a0:::benchmark_data_smithi102_461693_object41:head# 2024-04-25T09:26:41.076 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.093 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c6d379b1:::benchmark_data_smithi102_461693_object304:head# 2024-04-25T09:26:41.093 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.109 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c74b1c18:::benchmark_data_smithi102_461693_object113:head# 2024-04-25T09:26:41.110 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.126 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c7aec9f5:::benchmark_data_smithi102_467209_object18:head# 2024-04-25T09:26:41.126 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.143 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c7f88b4e:::benchmark_data_smithi102_461693_object284:head# 2024-04-25T09:26:41.143 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.160 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c7fe9f67:::benchmark_data_smithi102_467209_object43:head# 2024-04-25T09:26:41.160 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.176 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c88cd082:::benchmark_data_smithi102_461693_object259:head# 2024-04-25T09:26:41.177 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.227 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c8e2e872:::benchmark_data_smithi102_461693_object39:head# 2024-04-25T09:26:41.227 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.243 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c93a1b98:::benchmark_data_smithi102_461693_object142:head# 2024-04-25T09:26:41.243 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.260 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c986488b:::benchmark_data_smithi102_467209_object4:head# 2024-04-25T09:26:41.260 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.276 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c9c320e5:::benchmark_data_smithi102_461693_object249:head# 2024-04-25T09:26:41.276 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.293 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ca288c06:::benchmark_data_smithi102_461693_object153:head# 2024-04-25T09:26:41.293 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.310 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ca29967a:::benchmark_data_smithi102_467209_object144:head# 2024-04-25T09:26:41.310 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.326 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:caf11a5f:::benchmark_data_smithi102_467209_object111:head# 2024-04-25T09:26:41.326 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.343 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cba55ca9:::benchmark_data_smithi102_461693_object283:head# 2024-04-25T09:26:41.343 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.360 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cbd30596:::benchmark_data_smithi102_461693_object145:head# 2024-04-25T09:26:41.360 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.376 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cc022f04:::benchmark_data_smithi102_467209_object98:head# 2024-04-25T09:26:41.376 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.393 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cc2c6f61:::benchmark_data_smithi102_467209_object108:head# 2024-04-25T09:26:41.393 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.410 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cc3765a4:::benchmark_data_smithi102_461693_object0:head# 2024-04-25T09:26:41.410 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.426 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ce561cbb:::benchmark_data_smithi102_461693_object244:head# 2024-04-25T09:26:41.426 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.443 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cec1e000:::benchmark_data_smithi102_461693_object171:head# 2024-04-25T09:26:41.443 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.460 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cefbcfec:::benchmark_data_smithi102_467209_object28:head# 2024-04-25T09:26:41.460 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.476 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d07784f4:::benchmark_data_smithi102_461693_object107:head# 2024-04-25T09:26:41.477 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.493 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d1040409:::benchmark_data_smithi102_467209_object119:head# 2024-04-25T09:26:41.493 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.510 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d131843b:::benchmark_data_smithi102_461693_object119:head# 2024-04-25T09:26:41.510 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.526 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d1c66f30:::benchmark_data_smithi102_467209_object152:head# 2024-04-25T09:26:41.527 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.543 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d1f2e35b:::benchmark_data_smithi102_461693_object257:head# 2024-04-25T09:26:41.543 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.560 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d28830cc:::benchmark_data_smithi102_467209_object116:head# 2024-04-25T09:26:41.560 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.577 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d2bb2fe0:::benchmark_data_smithi102_461693_object197:head# 2024-04-25T09:26:41.577 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.593 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d2f5066b:::benchmark_data_smithi102_461693_object307:head# 2024-04-25T09:26:41.593 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.610 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d39e771b:::benchmark_data_smithi102_467209_object138:head# 2024-04-25T09:26:41.610 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.667 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d3a5ef94:::benchmark_data_smithi102_467209_object62:head# 2024-04-25T09:26:41.668 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.710 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d4b957d9:::benchmark_data_smithi102_467209_object102:head# 2024-04-25T09:26:41.710 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.727 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d7474635:::benchmark_data_smithi102_467209_object35:head# 2024-04-25T09:26:41.727 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.743 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d77f7f03:::benchmark_data_smithi102_461693_object170:head# 2024-04-25T09:26:41.743 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.760 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d95e4094:::benchmark_data_smithi102_461693_object105:head# 2024-04-25T09:26:41.760 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.777 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:da23c414:::benchmark_data_smithi102_461693_object57:head# 2024-04-25T09:26:41.777 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.793 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:da5b1f76:::benchmark_data_smithi102_461693_object73:head# 2024-04-25T09:26:41.794 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.810 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:da5b6865:::benchmark_data_smithi102_461693_object290:head# 2024-04-25T09:26:41.811 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.827 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:db2f70fa:::benchmark_data_smithi102_461693_object230:head# 2024-04-25T09:26:41.827 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.844 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:db345364:::benchmark_data_smithi102_467209_object115:head# 2024-04-25T09:26:41.844 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.860 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:db42a2eb:::benchmark_data_smithi102_467209_object97:head# 2024-04-25T09:26:41.860 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.877 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dbf5b9d1:::benchmark_data_smithi102_461693_object312:head# 2024-04-25T09:26:41.877 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.894 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dcecc90c:::benchmark_data_smithi102_467209_object23:head# 2024-04-25T09:26:41.894 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.910 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:de452fc5:::benchmark_data_smithi102_461693_object167:head# 2024-04-25T09:26:41.911 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.927 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:de96a9bf:::benchmark_data_smithi102_467209_object27:head# 2024-04-25T09:26:41.927 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.944 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e195d889:::benchmark_data_smithi102_461693_object129:head# 2024-04-25T09:26:41.944 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:41.960 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e2323390:::benchmark_data_smithi102_461693_object33:head# 2024-04-25T09:26:41.960 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.002 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e2b7382e:::benchmark_data_smithi102_461693_object210:head# 2024-04-25T09:26:42.002 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.019 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e2bbe377:::benchmark_data_smithi102_461693_object114:head# 2024-04-25T09:26:42.019 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.035 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e301fcee:::benchmark_data_smithi102_461693_object62:head# 2024-04-25T09:26:42.035 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.052 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e302dfe1:::benchmark_data_smithi102_461693_object227:head# 2024-04-25T09:26:42.052 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.069 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e4324c30:::benchmark_data_smithi102_461693_object158:head# 2024-04-25T09:26:42.069 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.085 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e4358590:::benchmark_data_smithi102_467209_object104:head# 2024-04-25T09:26:42.086 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.102 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e447aef9:::benchmark_data_smithi102_461693_object180:head# 2024-04-25T09:26:42.102 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.119 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e4558303:::benchmark_data_smithi102_461693_object4:head# 2024-04-25T09:26:42.119 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.177 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e5053873:::benchmark_data_smithi102_461693_object81:head# 2024-04-25T09:26:42.177 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.219 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e5210074:::benchmark_data_smithi102_461693_object3:head# 2024-04-25T09:26:42.219 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.280 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e6300266:::benchmark_data_smithi102_467209_object41:head# 2024-04-25T09:26:42.280 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.311 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e7382a9c:::benchmark_data_smithi102_461693_object299:head# 2024-04-25T09:26:42.311 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.327 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e7710cbe:::benchmark_data_smithi102_467209_object53:head# 2024-04-25T09:26:42.327 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.344 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e819bcc3:::benchmark_data_smithi102_461693_object84:head# 2024-04-25T09:26:42.344 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.361 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e84e09f2:::benchmark_data_smithi102_467209_object64:head# 2024-04-25T09:26:42.361 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.377 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e88ddb38:::benchmark_data_smithi102_467209_object128:head# 2024-04-25T09:26:42.377 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.394 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e8d6bf29:::benchmark_data_smithi102_461693_object101:head# 2024-04-25T09:26:42.394 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.411 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e9bbbf61:::benchmark_data_smithi102_467209_object93:head# 2024-04-25T09:26:42.411 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.427 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ea24f1cb:::benchmark_data_smithi102_461693_object186:head# 2024-04-25T09:26:42.427 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.444 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ec9bc09f:::benchmark_data_smithi102_461693_object229:head# 2024-04-25T09:26:42.444 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.461 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ecca2721:::benchmark_data_smithi102_461693_object134:head# 2024-04-25T09:26:42.461 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.477 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ecdd65cb:::benchmark_data_smithi102_461693_object269:head# 2024-04-25T09:26:42.478 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.494 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ed51570a:::benchmark_data_smithi102_467209_object80:head# 2024-04-25T09:26:42.494 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.536 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ee2ac7ee:::benchmark_data_smithi102_467209_object10:head# 2024-04-25T09:26:42.536 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.552 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ee986a84:::benchmark_data_smithi102_467209_object117:head# 2024-04-25T09:26:42.553 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.569 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:eed87a51:::benchmark_data_smithi102_461693_object191:head# 2024-04-25T09:26:42.570 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.586 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ef8637ba:::benchmark_data_smithi102_461693_object151:head# 2024-04-25T09:26:42.586 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.603 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f02b8d22:::benchmark_data_smithi102_461693_object168:head# 2024-04-25T09:26:42.603 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.619 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f06a39d9:::benchmark_data_smithi102_461693_object275:head# 2024-04-25T09:26:42.619 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.636 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f136e2f9:::benchmark_data_smithi102_461693_object267:head# 2024-04-25T09:26:42.636 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.653 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f19ac1c3:::benchmark_data_smithi102_467209_object31:head# 2024-04-25T09:26:42.653 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.669 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f2467763:::benchmark_data_smithi102_461693_object156:head# 2024-04-25T09:26:42.670 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.686 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f2e23aef:::benchmark_data_smithi102_467209_object33:head# 2024-04-25T09:26:42.686 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.703 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f313055e:::benchmark_data_smithi102_467209_object103:head# 2024-04-25T09:26:42.703 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.719 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f3685b12:::benchmark_data_smithi102_461693_object212:head# 2024-04-25T09:26:42.719 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.736 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f39374a5:::benchmark_data_smithi102_461693_object184:head# 2024-04-25T09:26:42.736 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.753 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f5c19b84:::benchmark_data_smithi102_461693_object225:head# 2024-04-25T09:26:42.753 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.769 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f5fbc496:::benchmark_data_smithi102_467209_object96:head# 2024-04-25T09:26:42.769 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.786 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f69416ea:::benchmark_data_smithi102_461693_object235:head# 2024-04-25T09:26:42.786 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.803 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f6d72824:::benchmark_data_smithi102_467209_object140:head# 2024-04-25T09:26:42.803 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.828 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f7295629:::benchmark_data_smithi102_461693_object87:head# 2024-04-25T09:26:42.828 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.845 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f8aeba6c:::benchmark_data_smithi102_467209_object148:head# 2024-04-25T09:26:42.845 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.861 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f90d383f:::benchmark_data_smithi102_461693_object203:head# 2024-04-25T09:26:42.862 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.878 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f939e8fc:::benchmark_data_smithi102_461693_object48:head# 2024-04-25T09:26:42.878 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.895 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f993a43e:::benchmark_data_smithi102_461693_object99:head# 2024-04-25T09:26:42.895 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.911 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f9fcdfe5:::benchmark_data_smithi102_467209_object52:head# 2024-04-25T09:26:42.912 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.928 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f9fdbd9c:::benchmark_data_smithi102_461693_object86:head# 2024-04-25T09:26:42.928 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.945 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fa2a1c9b:::benchmark_data_smithi102_461693_object47:head# 2024-04-25T09:26:42.945 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.962 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:facfae24:::benchmark_data_smithi102_467209_object89:head# 2024-04-25T09:26:42.962 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.978 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fb939772:::benchmark_data_smithi102_461693_object9:head# 2024-04-25T09:26:42.979 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:42.995 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fbd71e78:::benchmark_data_smithi102_467209_object90:head# 2024-04-25T09:26:42.995 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:43.012 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fca3284e:::benchmark_data_smithi102_461693_object187:head# 2024-04-25T09:26:43.012 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:43.029 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fe98a8ca:::benchmark_data_smithi102_461693_object258:head# 2024-04-25T09:26:43.029 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:43.045 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ff7040e0:::benchmark_data_smithi102_461693_object281:head# 2024-04-25T09:26:43.045 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:43.062 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ffab2f9e:::benchmark_data_smithi102_461693_object154:head# 2024-04-25T09:26:43.062 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:26:43.078 INFO:tasks.workunit.client.0.smithi102.stdout:write_pg epoch 225 info 1.0( v 225'337 (10'176,225'337] local-lis/les=24/25 n=481 ec=7/7 lis/c=24/24 les/c/f=25/25/0 sis=24) 2024-04-25T09:26:43.095 INFO:tasks.workunit.client.0.smithi102.stdout:Import successful 2024-04-25T09:26:43.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:157: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:26:43.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:26:43.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:26:43.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:26:43.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:26:43.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:26:43.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:26:43.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:26:43.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:26:43.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:26:43.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:26:43.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:26:43.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:26:43.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:26:43.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:26:43.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:26:43.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:26:43.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:26:43.635 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:26:43.636 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:26:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:26:43.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:26:43.638 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:26:43.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:26:43.639 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:26:43.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:26:43.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:26:43.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:26:43.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:26:43.673 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:43.685+0000 7f3d1c04d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:43.675 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:43.689+0000 7f3d1c04d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:43.677 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:43.689+0000 7f3d1c04d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:44.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:26:44.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:26:44.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:26:44.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:26:44.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:26:44.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:44.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:26:44.052 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:26:44.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:44.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:44.224 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:44.236+0000 7f3d1c04d7c0 -1 Falling back to public interface 2024-04-25T09:26:44.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:45.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:45.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:45.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:26:45.481 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:26:45.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:45.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:45.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:46.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:46.903 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:26:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:26:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:46.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:47.001 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:47.012+0000 7f3d1c04d7c0 -1 osd.0 229 log_to_monitors true 2024-04-25T09:26:47.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:48.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:48.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:48.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:26:48.401 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:26:48.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:48.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:48.823 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 232 up_thru 232 down_at 230 last_clean_interval [24,229) [v2:127.0.0.1:6802/4089610000,v1:127.0.0.1:6803/4089610000] [v2:127.0.0.1:6804/4089610000,v1:127.0.0.1:6805/4089610000] exists,up 150f6751-a5c0-42ad-b859-7e7a4382302c 2024-04-25T09:26:48.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:26:48.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:26:48.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:26:48.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:159: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:26:48.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:26:48.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:26:48.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:26:48.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:26:48.825 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:26:48.825 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:26:48.825 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:26:48.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:26:48.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:26:48.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:26:48.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:26:48.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:26:48.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:26:48.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:26:48.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:26:49.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:26:49.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:26:49.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:26:49.356 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:26:49.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=996432412675 2024-04-25T09:26:49.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 996432412675 2024-04-25T09:26:49.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-996432412675' 2024-04-25T09:26:49.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:26:49.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-996432412675 2024-04-25T09:26:49.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:26:49.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:26:49.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-996432412675 2024-04-25T09:26:49.531 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:26:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=996432412675 2024-04-25T09:26:49.532 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 996432412675 2024-04-25T09:26:49.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 996432412675' 2024-04-25T09:26:49.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:26:49.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 996432412675 -lt 996432412675 2024-04-25T09:26:49.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:26:49.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:26:49.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:26:50.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:26:50.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:26:50.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:26:50.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:26:50.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:26:50.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:26:50.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:26:50.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:26:50.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:26:50.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:26:50.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:26:50.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:26:51.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:26:51.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:26:51.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:26:51.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:26:51.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:26:51.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:26:51.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:26:51.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:26:51.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:26:51.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:26:51.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:26:51.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:26:51.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:26:51.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:26:51.645 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:26:51.645 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:26:51.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:26:51.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:26:51.647 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:26:51.647 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:51.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:26:51.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:26:51.648 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:51.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:26:51.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:26:51.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:26:51.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:26:51.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:51.667 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:26:51.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.450371 2024-04-25T09:26:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:26:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:26:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:26:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:26:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:26:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:26:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:26:51.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:26:51.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:26:51.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:26:51.670 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:26:51.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:26:51.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:26:51.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:26:51.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:26:51.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:26:51.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:26:51.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:26:51.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:26:51.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:26:51.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:26:51.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:51.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:26:51.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:26:51.675 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:51.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:26:51.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:26:51.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:26:51.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:26:51.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:51.678 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:26:51.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.450371 2024-04-25T09:26:51.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:26:51.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:26:51.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:26:51.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:26:51.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:51.680 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:26:51.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.450371 2024-04-25T09:26:51.682 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:26:51.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:26:51.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:26:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:26:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_split td/pg-split-merge 2024-04-25T09:26:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:163: TEST_import_after_split: local dir=td/pg-split-merge 2024-04-25T09:26:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:165: TEST_import_after_split: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:26:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:26:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:26:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:26:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:26:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:26:51.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:26:52.170 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:26:52.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:26:52.171 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:26:52.171 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:26:52.171 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:52.171 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:26:52.172 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:26:52.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:26:52.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:26:52.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:26:52.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:26:52.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:26:52.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:26:52.424 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:26:52.424 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:26:52.424 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:26:52.424 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:26:52.425 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:26:52.425 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:52.425 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:26:52.425 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.450371/ceph-mon.a.asok 2024-04-25T09:26:52.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:26:52.425 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.450371/ceph-mon.a.asok config get fsid 2024-04-25T09:26:52.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:26:52.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:26:52.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:26:52.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:26:52.535 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:26:52.536 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:26:52.536 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:26:52.536 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:26:52.537 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:26:52.537 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:52.537 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:26:52.537 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.450371/ceph-mon.a.asok 2024-04-25T09:26:52.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:26:52.537 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.450371/ceph-mon.a.asok config get mon_host 2024-04-25T09:26:52.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:166: TEST_import_after_split: run_mgr td/pg-split-merge x 2024-04-25T09:26:52.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:26:52.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:26:52.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:26:52.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:26:52.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:26:52.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:26:53.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:26:53.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:26:53.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:26:53.559 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:26:53.559 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:53.559 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:26:53.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:26:53.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:26:53.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:26:53.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:167: TEST_import_after_split: run_osd td/pg-split-merge 0 2024-04-25T09:26:53.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:26:53.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:26:53.596 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:26:53.597 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:26:53.598 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:26:53.598 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:53.598 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:26:53.598 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:26:53.599 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:26:53.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b7b813c4-be34-4c65-abf1-0e52eb14287b 2024-04-25T09:26:53.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b7b813c4-be34-4c65-abf1-0e52eb14287b' 2024-04-25T09:26:53.600 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 b7b813c4-be34-4c65-abf1-0e52eb14287b 2024-04-25T09:26:53.601 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:26:53.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDdISpm7FK9JRAAeM0iotD+SFazzfAcb5sIVQ== 2024-04-25T09:26:53.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDdISpm7FK9JRAAeM0iotD+SFazzfAcb5sIVQ=="}' 2024-04-25T09:26:53.617 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b7b813c4-be34-4c65-abf1-0e52eb14287b -i td/pg-split-merge/0/new.json 2024-04-25T09:26:53.925 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:26:53.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:26:53.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDdISpm7FK9JRAAeM0iotD+SFazzfAcb5sIVQ== --osd-uuid b7b813c4-be34-4c65-abf1-0e52eb14287b 2024-04-25T09:26:53.979 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:53.992+0000 7f020bf607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:53.993 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:54.004+0000 7f020bf607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:53.994 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:54.008+0000 7f020bf607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:53.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:54.008+0000 7f020bf607c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:26:53.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:54.008+0000 7f020bf607c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:26:53.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:54.008+0000 7f020bf607c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:26:54.025 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:26:54.036+0000 7f020bf607c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:27:00.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:27:00.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:27:00.389 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:27:00.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:27:00.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:27:00.952 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:27:00.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:27:00.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:27:00.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:27:00.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:27:00.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:27:00.986 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:27:01.000+0000 7fe44d1a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:00.988 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:27:01.000+0000 7fe44d1a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:00.989 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:27:01.000+0000 7fe44d1a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:01.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:27:01.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:27:01.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:27:01.433 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:27:01.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:27:01.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:27:01.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:01.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:27:01.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:01.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:01.560 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:27:01.572+0000 7fe44d1a17c0 -1 Falling back to public interface 2024-04-25T09:27:01.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:02.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:02.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:02.854 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:27:02.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:27:02.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:02.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:03.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:04.175 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:27:04.188+0000 7fe44d1a17c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:27:04.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:04.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:04.261 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:27:04.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:27:04.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:04.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:04.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:05.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:05.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:05.888 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:27:05.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:27:05.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:05.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:06.406 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:07.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:07.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:07.408 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:27:07.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:27:07.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:07.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:07.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:08.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:08.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:08.837 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:27:08.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:27:08.837 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:08.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:09.367 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:10.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:10.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:10.370 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:27:10.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:27:10.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:10.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:10.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:11.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:11.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:11.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:27:11.786 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:27:11.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:11.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:12.313 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:13.315 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:13.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:13.316 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:27:13.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:27:13.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:13.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:13.782 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:14.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:14.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:14.783 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:27:14.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:27:14.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:14.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:15.330 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:16.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:16.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:16.332 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:27:16.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:27:16.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:16.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:16.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:17.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:17.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:17.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:27:17.755 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:27:17.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:17.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:18.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:19.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:19.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:19.893 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:27:19.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:27:19.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:19.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:20.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:21.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:21.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:21.538 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:27:21.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:27:21.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:21.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:22.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:23.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:23.256 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:27:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:27:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:23.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:23.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:24.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:24.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:24.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:27:24.993 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:27:24.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:24.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:25.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:26.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:26.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:26.853 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:27:26.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:27:26.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:26.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:28.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:29.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:29.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:29.056 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:27:29.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:27:29.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:29.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:29.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:30.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:30.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:30.578 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:27:30.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:27:30.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:30.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:30.993 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3522979535,v1:127.0.0.1:6803/3522979535] [v2:127.0.0.1:6804/3522979535,v1:127.0.0.1:6805/3522979535] exists,up b7b813c4-be34-4c65-abf1-0e52eb14287b 2024-04-25T09:27:30.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:27:30.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:27:30.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:27:30.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:169: TEST_import_after_split: ceph osd pool create foo 1 2024-04-25T09:27:31.572 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'foo' already exists 2024-04-25T09:27:31.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:170: TEST_import_after_split: wait_for_clean 2024-04-25T09:27:31.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:27:31.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:27:31.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:27:31.589 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:27:31.589 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:27:31.589 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:27:31.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:27:31.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:27:31.590 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:27:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:27:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:27:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:27:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:27:31.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:27:31.685 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:27:32.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:27:32.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:27:32.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:27:32.259 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:27:32.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836485 2024-04-25T09:27:32.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836485 2024-04-25T09:27:32.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836485' 2024-04-25T09:27:32.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:27:32.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836485 2024-04-25T09:27:32.498 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:27:32.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:27:32.500 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836485 2024-04-25T09:27:32.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:27:32.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836485 2024-04-25T09:27:32.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-04-25T09:27:32.502 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836485 2024-04-25T09:27:32.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:27:32.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836483 -lt 21474836485 2024-04-25T09:27:32.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:27:33.928 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:27:33.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:27:34.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836485 -lt 21474836485 2024-04-25T09:27:34.398 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:27:34.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:27:34.399 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:27:34.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:27:34.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:27:34.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:27:34.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:27:34.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:27:34.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:27:34.930 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:27:34.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:27:35.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:27:35.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:27:35.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:27:35.389 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:27:35.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:27:35.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:27:35.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:27:35.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:171: TEST_import_after_split: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-04-25T09:27:36.021 INFO:tasks.workunit.client.0.smithi102.stdout:hints = 1 2024-04-25T09:27:36.022 INFO:tasks.workunit.client.0.smithi102.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-04-25T09:27:36.022 INFO:tasks.workunit.client.0.smithi102.stdout:Object prefix: benchmark_data_smithi102_474039 2024-04-25T09:27:36.022 INFO:tasks.workunit.client.0.smithi102.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:27:36.022 INFO:tasks.workunit.client.0.smithi102.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:27:37.021 INFO:tasks.workunit.client.0.smithi102.stdout: 1 16 112 96 0.0937386 0.09375 0.13435 0.159255 2024-04-25T09:27:38.022 INFO:tasks.workunit.client.0.smithi102.stdout: 2 16 224 208 0.101547 0.109375 0.133806 0.148706 2024-04-25T09:27:39.022 INFO:tasks.workunit.client.0.smithi102.stdout: 3 16 321 305 0.0992687 0.0947266 0.159126 0.152351 2024-04-25T09:27:40.022 INFO:tasks.workunit.client.0.smithi102.stdout:Total time run: 3.08382 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Total writes made: 321 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Write size: 1024 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Object size: 1024 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Bandwidth (MB/sec): 0.101652 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Bandwidth: 0.00875282 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Max bandwidth (MB/sec): 0.109375 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Min bandwidth (MB/sec): 0.09375 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Average IOPS: 104 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev IOPS: 8.96289 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Max IOPS: 112 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Min IOPS: 96 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Average Latency(s): 0.150608 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Stddev Latency(s): 0.0266786 2024-04-25T09:27:40.023 INFO:tasks.workunit.client.0.smithi102.stdout:Max latency(s): 0.217872 2024-04-25T09:27:40.024 INFO:tasks.workunit.client.0.smithi102.stdout:Min latency(s): 0.103496 2024-04-25T09:27:40.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:173: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:27:40.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:27:40.087 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:27:40.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:27:40.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:27:40.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:27:40.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:27:40.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:174: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:27:40.595 INFO:tasks.workunit.client.0.smithi102.stderr:Exporting 1.0 info 1.0( v 9'322 (0'0,9'322] local-lis/les=7/8 n=322 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:27:40.607 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:00100e1f:::benchmark_data_smithi102_474039_object285:head# 2024-04-25T09:27:40.608 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:016d36bd:::benchmark_data_smithi102_474039_object248:head# 2024-04-25T09:27:40.608 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:02ab28e4:::benchmark_data_smithi102_474039_object287:head# 2024-04-25T09:27:40.608 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:02fc2004:::benchmark_data_smithi102_474039_object307:head# 2024-04-25T09:27:40.609 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:030f3311:::benchmark_data_smithi102_474039_object306:head# 2024-04-25T09:27:40.609 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:034cce74:::benchmark_data_smithi102_474039_object207:head# 2024-04-25T09:27:40.611 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:037b8264:::benchmark_data_smithi102_474039_object19:head# 2024-04-25T09:27:40.611 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0473b718:::benchmark_data_smithi102_474039_object220:head# 2024-04-25T09:27:40.611 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0508c379:::benchmark_data_smithi102_474039_object162:head# 2024-04-25T09:27:40.611 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:064ec71f:::benchmark_data_smithi102_474039_object71:head# 2024-04-25T09:27:40.612 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:06f42a1b:::benchmark_data_smithi102_474039_object263:head# 2024-04-25T09:27:40.612 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:07db957e:::benchmark_data_smithi102_474039_object98:head# 2024-04-25T09:27:40.612 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:07e0f0ab:::benchmark_data_smithi102_474039_object291:head# 2024-04-25T09:27:40.613 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:087fa2a2:::benchmark_data_smithi102_474039_object91:head# 2024-04-25T09:27:40.613 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:091e887c:::benchmark_data_smithi102_474039_object53:head# 2024-04-25T09:27:40.614 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0a42931d:::benchmark_data_smithi102_474039_object301:head# 2024-04-25T09:27:40.615 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0a4a118b:::benchmark_data_smithi102_474039_object223:head# 2024-04-25T09:27:40.615 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0af76ace:::benchmark_data_smithi102_474039_object262:head# 2024-04-25T09:27:40.615 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0b71271b:::benchmark_data_smithi102_474039_object312:head# 2024-04-25T09:27:40.615 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0baa67d4:::benchmark_data_smithi102_474039_object171:head# 2024-04-25T09:27:40.615 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0cd0abf2:::benchmark_data_smithi102_474039_object14:head# 2024-04-25T09:27:40.631 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0d156d14:::benchmark_data_smithi102_474039_object313:head# 2024-04-25T09:27:40.632 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0dfdd378:::benchmark_data_smithi102_474039_object295:head# 2024-04-25T09:27:40.633 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0ec0b5ce:::benchmark_data_smithi102_474039_object204:head# 2024-04-25T09:27:40.635 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0ed0639c:::benchmark_data_smithi102_474039_object136:head# 2024-04-25T09:27:40.636 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0f5eb555:::benchmark_data_smithi102_474039_object80:head# 2024-04-25T09:27:40.636 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0f9cd47b:::benchmark_data_smithi102_474039_object232:head# 2024-04-25T09:27:40.637 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:0fad4256:::benchmark_data_smithi102_474039_object113:head# 2024-04-25T09:27:40.637 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:119d5994:::benchmark_data_smithi102_474039_object280:head# 2024-04-25T09:27:40.638 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:12683126:::benchmark_data_smithi102_474039_object82:head# 2024-04-25T09:27:40.638 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:12b6f85b:::benchmark_data_smithi102_474039_object151:head# 2024-04-25T09:27:40.639 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:15002f77:::benchmark_data_smithi102_474039_object34:head# 2024-04-25T09:27:40.639 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:168415d3:::benchmark_data_smithi102_474039_object5:head# 2024-04-25T09:27:40.640 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:16baadaa:::benchmark_data_smithi102_474039_object175:head# 2024-04-25T09:27:40.641 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:18143727:::benchmark_data_smithi102_474039_object128:head# 2024-04-25T09:27:40.642 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1845f65a:::benchmark_data_smithi102_474039_object217:head# 2024-04-25T09:27:40.642 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1858dfac:::benchmark_data_smithi102_474039_object269:head# 2024-04-25T09:27:40.643 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:18708f85:::benchmark_data_smithi102_474039_object278:head# 2024-04-25T09:27:40.643 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:19d44365:::benchmark_data_smithi102_474039_object56:head# 2024-04-25T09:27:40.644 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:19eb82fa:::benchmark_data_smithi102_474039_object108:head# 2024-04-25T09:27:40.644 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1a051d39:::benchmark_data_smithi102_474039_object311:head# 2024-04-25T09:27:40.644 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1a9350f4:::benchmark_data_smithi102_474039_object86:head# 2024-04-25T09:27:40.645 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1a9d0683:::benchmark_data_smithi102_474039_object63:head# 2024-04-25T09:27:40.645 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1b8bb510:::benchmark_data_smithi102_474039_object302:head# 2024-04-25T09:27:40.646 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1bfa1058:::benchmark_data_smithi102_474039_object117:head# 2024-04-25T09:27:40.646 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1e1f8f1c:::benchmark_data_smithi102_474039_object203:head# 2024-04-25T09:27:40.647 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:1f50317b:::benchmark_data_smithi102_474039_object42:head# 2024-04-25T09:27:40.647 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:20c4c6a3:::benchmark_data_smithi102_474039_object95:head# 2024-04-25T09:27:40.648 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:218a2cb2:::benchmark_data_smithi102_474039_object282:head# 2024-04-25T09:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:242d6fe2:::benchmark_data_smithi102_474039_object158:head# 2024-04-25T09:27:40.649 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:25e1ed0c:::benchmark_data_smithi102_474039_object137:head# 2024-04-25T09:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:272b798a:::benchmark_data_smithi102_474039_object100:head# 2024-04-25T09:27:40.650 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:283127ae:::benchmark_data_smithi102_474039_object193:head# 2024-04-25T09:27:40.651 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:288243f9:::benchmark_data_smithi102_474039_object153:head# 2024-04-25T09:27:40.652 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:28cc675e:::benchmark_data_smithi102_474039_object318:head# 2024-04-25T09:27:40.652 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2c04df7f:::benchmark_data_smithi102_474039_object79:head# 2024-04-25T09:27:40.653 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:2dda7f96:::benchmark_data_smithi102_474039_object144:head# 2024-04-25T09:27:40.653 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:31df5474:::benchmark_data_smithi102_474039_object283:head# 2024-04-25T09:27:40.653 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:32997b14:::benchmark_data_smithi102_474039_object22:head# 2024-04-25T09:27:40.654 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3362a17e:::benchmark_data_smithi102_474039_object74:head# 2024-04-25T09:27:40.666 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:356c603e:::benchmark_data_smithi102_474039_object221:head# 2024-04-25T09:27:40.674 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:366c2233:::benchmark_data_smithi102_474039_object10:head# 2024-04-25T09:27:40.675 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:36d21c33:::benchmark_data_smithi102_474039_object205:head# 2024-04-25T09:27:40.692 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:37084488:::benchmark_data_smithi102_474039_object190:head# 2024-04-25T09:27:40.708 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3713e5dc:::benchmark_data_smithi102_474039_object69:head# 2024-04-25T09:27:40.708 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:389d97cb:::benchmark_data_smithi102_474039_object274:head# 2024-04-25T09:27:40.725 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3a5a88b7:::benchmark_data_smithi102_474039_object26:head# 2024-04-25T09:27:40.733 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3b32d00d:::benchmark_data_smithi102_474039_object219:head# 2024-04-25T09:27:40.741 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3c35bebb:::benchmark_data_smithi102_474039_object159:head# 2024-04-25T09:27:40.749 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3c7f614f:::benchmark_data_smithi102_474039_object7:head# 2024-04-25T09:27:40.750 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3d2b26a7:::benchmark_data_smithi102_474039_object17:head# 2024-04-25T09:27:40.750 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3d471739:::benchmark_data_smithi102_474039_object97:head# 2024-04-25T09:27:40.751 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3e0c45d9:::benchmark_data_smithi102_474039_object51:head# 2024-04-25T09:27:40.751 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:3e52bb25:::benchmark_data_smithi102_474039_object258:head# 2024-04-25T09:27:40.752 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:412fb915:::benchmark_data_smithi102_474039_object156:head# 2024-04-25T09:27:40.753 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4170f76c:::benchmark_data_smithi102_474039_object143:head# 2024-04-25T09:27:40.754 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4284d7d2:::benchmark_data_smithi102_474039_object11:head# 2024-04-25T09:27:40.754 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4362be4d:::benchmark_data_smithi102_474039_object145:head# 2024-04-25T09:27:40.755 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:43989524:::benchmark_data_smithi102_474039_object16:head# 2024-04-25T09:27:40.755 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:455fdb9c:::benchmark_data_smithi102_474039_object88:head# 2024-04-25T09:27:40.756 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:47245629:::benchmark_data_smithi102_474039_object66:head# 2024-04-25T09:27:40.756 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4782c34d:::benchmark_data_smithi102_474039_object180:head# 2024-04-25T09:27:40.757 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:493b10f5:::benchmark_data_smithi102_474039_object236:head# 2024-04-25T09:27:40.757 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:49a2c41e:::benchmark_data_smithi102_474039_object125:head# 2024-04-25T09:27:40.758 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:49bfbf8f:::benchmark_data_smithi102_474039_object111:head# 2024-04-25T09:27:40.759 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:49d2acb0:::benchmark_data_smithi102_474039_object84:head# 2024-04-25T09:27:40.759 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4ada50a6:::benchmark_data_smithi102_474039_object109:head# 2024-04-25T09:27:40.760 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4b4086e5:::benchmark_data_smithi102_474039_object275:head# 2024-04-25T09:27:40.760 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4d2b067d:::benchmark_data_smithi102_474039_object256:head# 2024-04-25T09:27:40.761 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4e4a078b:::benchmark_data_smithi102_474039_object102:head# 2024-04-25T09:27:40.762 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4e98afd2:::benchmark_data_smithi102_474039_object229:head# 2024-04-25T09:27:40.762 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4ebffa2f:::benchmark_data_smithi102_474039_object251:head# 2024-04-25T09:27:40.763 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4ec71e53:::benchmark_data_smithi102_474039_object183:head# 2024-04-25T09:27:40.764 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:4ff94952:::benchmark_data_smithi102_474039_object6:head# 2024-04-25T09:27:40.765 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5121d7ac:::benchmark_data_smithi102_474039_object90:head# 2024-04-25T09:27:40.765 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:51a8b164:::benchmark_data_smithi102_474039_object233:head# 2024-04-25T09:27:40.765 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5303b8d7:::benchmark_data_smithi102_474039_object4:head# 2024-04-25T09:27:40.766 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:560fc05e:::benchmark_data_smithi102_474039_object246:head# 2024-04-25T09:27:40.766 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:58166439:::benchmark_data_smithi102_474039_object32:head# 2024-04-25T09:27:40.767 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5881a447:::benchmark_data_smithi102_474039_object35:head# 2024-04-25T09:27:40.767 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:58f91e48:::benchmark_data_smithi102_474039_object103:head# 2024-04-25T09:27:40.768 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:59ab0008:::benchmark_data_smithi102_474039_object68:head# 2024-04-25T09:27:40.768 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:59b4e707:::benchmark_data_smithi102_474039_object284:head# 2024-04-25T09:27:40.769 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:59b7abdf:::benchmark_data_smithi102_474039_object186:head# 2024-04-25T09:27:40.770 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5b0b0b70:::benchmark_data_smithi102_474039_object168:head# 2024-04-25T09:27:40.770 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5cc9558d:::benchmark_data_smithi102_474039_object201:head# 2024-04-25T09:27:40.771 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5cf45479:::benchmark_data_smithi102_474039_object316:head# 2024-04-25T09:27:40.771 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5d7d41eb:::benchmark_data_smithi102_474039_object202:head# 2024-04-25T09:27:40.772 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5d8c345c:::benchmark_data_smithi102_474039_object317:head# 2024-04-25T09:27:40.773 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5dca7eef:::benchmark_data_smithi102_474039_object195:head# 2024-04-25T09:27:40.774 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5e27b06c:::benchmark_data_smithi102_474039_object154:head# 2024-04-25T09:27:40.774 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:5f003aea:::benchmark_data_smithi102_474039_object122:head# 2024-04-25T09:27:40.775 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:60982b13:::benchmark_data_smithi102_474039_object27:head# 2024-04-25T09:27:40.775 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:610cfa39:::benchmark_data_smithi102_474039_object118:head# 2024-04-25T09:27:40.776 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:617982b8:::benchmark_data_smithi102_474039_object277:head# 2024-04-25T09:27:40.777 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:61fe964f:::benchmark_data_smithi102_474039_object210:head# 2024-04-25T09:27:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6235c3e6:::benchmark_data_smithi102_474039_object189:head# 2024-04-25T09:27:40.778 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:638377f6:::benchmark_data_smithi102_474039_object177:head# 2024-04-25T09:27:40.779 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:638df456:::benchmark_data_smithi102_474039_object104:head# 2024-04-25T09:27:40.780 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:64248811:::benchmark_data_smithi102_474039_object155:head# 2024-04-25T09:27:40.780 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:677c0aaa:::benchmark_data_smithi102_474039_object227:head# 2024-04-25T09:27:40.781 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:67a2c881:::benchmark_data_smithi102_474039_object39:head# 2024-04-25T09:27:40.781 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:67ebb9ef:::benchmark_data_smithi102_474039_object148:head# 2024-04-25T09:27:40.782 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:68764934:::benchmark_data_smithi102_474039_object255:head# 2024-04-25T09:27:40.783 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6944e27b:::benchmark_data_smithi102_474039_object60:head# 2024-04-25T09:27:40.784 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:69b1a42e:::benchmark_data_smithi102_474039_object218:head# 2024-04-25T09:27:40.784 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6a07e9f8:::benchmark_data_smithi102_474039_object28:head# 2024-04-25T09:27:40.785 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6a080ceb:::benchmark_data_smithi102_474039_object40:head# 2024-04-25T09:27:40.785 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6a418838:::benchmark_data_smithi102_474039_object250:head# 2024-04-25T09:27:40.786 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6a91d9dc:::benchmark_data_smithi102_474039_object208:head# 2024-04-25T09:27:40.787 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6afabd1c:::benchmark_data_smithi102_474039_object140:head# 2024-04-25T09:27:40.787 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6c5b143e:::benchmark_data_smithi102_474039_object270:head# 2024-04-25T09:27:40.788 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6c82857d:::benchmark_data_smithi102_474039_object226:head# 2024-04-25T09:27:40.789 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6cce6c95:::benchmark_data_smithi102_474039_object49:head# 2024-04-25T09:27:40.789 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:6ea142d3:::benchmark_data_smithi102_474039_object41:head# 2024-04-25T09:27:40.790 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7023b0bd:::benchmark_data_smithi102_474039_object124:head# 2024-04-25T09:27:40.790 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:706481cb:::benchmark_data_smithi102_474039_object298:head# 2024-04-25T09:27:40.791 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:716b84ca:::benchmark_data_smithi102_474039_object165:head# 2024-04-25T09:27:40.792 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:71dad963:::benchmark_data_smithi102_474039_object130:head# 2024-04-25T09:27:40.793 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7298df79:::benchmark_data_smithi102_474039_object72:head# 2024-04-25T09:27:40.793 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:74627db4:::benchmark_data_smithi102_474039_object238:head# 2024-04-25T09:27:40.794 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:747bef36:::benchmark_data_smithi102_474039_object146:head# 2024-04-25T09:27:40.795 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:74c700d3:::benchmark_data_smithi102_474039_object200:head# 2024-04-25T09:27:40.795 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:756f95d3:::benchmark_data_smithi102_474039_object127:head# 2024-04-25T09:27:40.796 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7619e673:::benchmark_data_smithi102_474039_object94:head# 2024-04-25T09:27:40.796 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:76449db9:::benchmark_data_smithi102_474039_object141:head# 2024-04-25T09:27:40.797 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7867171d:::benchmark_data_smithi102_474039_object92:head# 2024-04-25T09:27:40.798 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:791732fc:::benchmark_data_smithi102_474039_object89:head# 2024-04-25T09:27:40.798 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:792876a9:::benchmark_data_smithi102_474039_object271:head# 2024-04-25T09:27:40.799 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7a8c047a:::benchmark_data_smithi102_474039_object126:head# 2024-04-25T09:27:40.800 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7b184e43:::benchmark_data_smithi102_474039_object52:head# 2024-04-25T09:27:40.800 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7bddf6f7:::benchmark_data_smithi102_474039_object132:head# 2024-04-25T09:27:40.801 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:7df96f0f:::benchmark_data_smithi102_474039_object15:head# 2024-04-25T09:27:40.801 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:80328a68:::benchmark_data_smithi102_474039_object247:head# 2024-04-25T09:27:40.802 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:80668dd0:::benchmark_data_smithi102_474039_object169:head# 2024-04-25T09:27:40.803 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:80c889ae:::benchmark_data_smithi102_474039_object43:head# 2024-04-25T09:27:40.803 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8278fb4f:::benchmark_data_smithi102_474039_object289:head# 2024-04-25T09:27:40.804 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:831dc679:::benchmark_data_smithi102_474039_object36:head# 2024-04-25T09:27:40.805 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:84902432:::benchmark_data_smithi102_474039_object24:head# 2024-04-25T09:27:40.805 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:86592b4a:::benchmark_data_smithi102_474039_object260:head# 2024-04-25T09:27:40.806 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:87a9131c:::benchmark_data_smithi102_474039_object244:head# 2024-04-25T09:27:40.807 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:87e3ba06:::benchmark_data_smithi102_474039_object268:head# 2024-04-25T09:27:40.807 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:88267fa3:::benchmark_data_smithi102_474039_object116:head# 2024-04-25T09:27:40.808 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:88706ab9:::benchmark_data_smithi102_474039_object59:head# 2024-04-25T09:27:40.808 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:88a98f2c:::benchmark_data_smithi102_474039_object8:head# 2024-04-25T09:27:40.809 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:899ddbe7:::benchmark_data_smithi102_474039_object170:head# 2024-04-25T09:27:40.810 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8c31299f:::benchmark_data_smithi102_474039_object142:head# 2024-04-25T09:27:40.810 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8c659197:::benchmark_data_smithi102_474039_object305:head# 2024-04-25T09:27:40.811 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8d5bcc76:::benchmark_data_smithi102_474039_object164:head# 2024-04-25T09:27:40.812 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8de977a4:::benchmark_data_smithi102_474039_object276:head# 2024-04-25T09:27:40.812 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8e54a6f0:::benchmark_data_smithi102_474039_object123:head# 2024-04-25T09:27:40.813 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8eb9693c:::benchmark_data_smithi102_474039_object299:head# 2024-04-25T09:27:40.814 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8f268b7b:::benchmark_data_smithi102_474039_object163:head# 2024-04-25T09:27:40.814 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:8fb5dd81:::benchmark_data_smithi102_474039_object31:head# 2024-04-25T09:27:40.815 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9015d089:::benchmark_data_smithi102_474039_object292:head# 2024-04-25T09:27:40.816 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9017a08a:::benchmark_data_smithi102_474039_object212:head# 2024-04-25T09:27:40.816 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:90930b9a:::benchmark_data_smithi102_474039_object46:head# 2024-04-25T09:27:40.817 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:91c76f76:::benchmark_data_smithi102_474039_object188:head# 2024-04-25T09:27:40.817 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9250853e:::benchmark_data_smithi102_474039_object265:head# 2024-04-25T09:27:40.818 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:92914a1f:::benchmark_data_smithi102_474039_object231:head# 2024-04-25T09:27:40.818 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:934cbb72:::benchmark_data_smithi102_474039_object75:head# 2024-04-25T09:27:40.819 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:947c7936:::benchmark_data_smithi102_474039_object110:head# 2024-04-25T09:27:40.819 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9485f46d:::benchmark_data_smithi102_474039_object290:head# 2024-04-25T09:27:40.820 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:94b492cf:::benchmark_data_smithi102_474039_object272:head# 2024-04-25T09:27:40.821 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9549b256:::benchmark_data_smithi102_474039_object9:head# 2024-04-25T09:27:40.822 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:95e884e1:::benchmark_data_smithi102_474039_object249:head# 2024-04-25T09:27:40.822 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9621544e:::benchmark_data_smithi102_474039_object279:head# 2024-04-25T09:27:40.823 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:965f10ca:::benchmark_data_smithi102_474039_object83:head# 2024-04-25T09:27:40.823 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:97115da6:::benchmark_data_smithi102_474039_object85:head# 2024-04-25T09:27:40.824 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:97e3beb2:::benchmark_data_smithi102_474039_object213:head# 2024-04-25T09:27:40.825 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:98f17a70:::benchmark_data_smithi102_474039_object45:head# 2024-04-25T09:27:40.825 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:99bb20a2:::benchmark_data_smithi102_474039_object73:head# 2024-04-25T09:27:40.825 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:99c72720:::benchmark_data_smithi102_474039_object211:head# 2024-04-25T09:27:40.826 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9a3605e2:::benchmark_data_smithi102_474039_object230:head# 2024-04-25T09:27:40.826 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9a896a36:::benchmark_data_smithi102_474039_object93:head# 2024-04-25T09:27:40.827 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9be82daf:::benchmark_data_smithi102_474039_object191:head# 2024-04-25T09:27:40.827 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9ceb25b1:::benchmark_data_smithi102_474039_object253:head# 2024-04-25T09:27:40.828 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9d1ff8ff:::benchmark_data_smithi102_474039_object194:head# 2024-04-25T09:27:40.829 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9d4981d9:::benchmark_data_smithi102_474039_object167:head# 2024-04-25T09:27:40.829 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9dbd761d:::benchmark_data_smithi102_474039_object54:head# 2024-04-25T09:27:40.830 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9dd01cce:::benchmark_data_smithi102_474039_object178:head# 2024-04-25T09:27:40.830 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9df2ece6:::benchmark_data_smithi102_474039_object78:head# 2024-04-25T09:27:40.831 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:9ed71fdd:::benchmark_data_smithi102_474039_object29:head# 2024-04-25T09:27:40.831 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a0621afc:::benchmark_data_smithi102_474039_object133:head# 2024-04-25T09:27:40.832 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a11c251c:::benchmark_data_smithi102_474039_object293:head# 2024-04-25T09:27:40.833 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a193ff46:::benchmark_data_smithi102_474039_object314:head# 2024-04-25T09:27:40.833 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a1d05115:::benchmark_data_smithi102_474039_object296:head# 2024-04-25T09:27:40.834 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a1e17490:::benchmark_data_smithi102_474039_object2:head# 2024-04-25T09:27:40.836 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a2467ccf:::benchmark_data_smithi102_474039_object23:head# 2024-04-25T09:27:40.836 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a3caa411:::benchmark_data_smithi102_474039_object99:head# 2024-04-25T09:27:40.836 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a3d45b93:::benchmark_data_smithi102_474039_object96:head# 2024-04-25T09:27:40.837 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a4ccd216:::benchmark_data_smithi102_474039_object319:head# 2024-04-25T09:27:40.837 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a70e57d8:::benchmark_data_smithi102_474039_object300:head# 2024-04-25T09:27:40.838 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a83cc239:::benchmark_data_smithi102_474039_object138:head# 2024-04-25T09:27:40.839 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a88493d3:::benchmark_data_smithi102_474039_object50:head# 2024-04-25T09:27:40.839 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:a9fa51a1:::benchmark_data_smithi102_474039_object241:head# 2024-04-25T09:27:40.840 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac1cc923:::benchmark_data_smithi102_474039_object288:head# 2024-04-25T09:27:40.841 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac6944ff:::benchmark_data_smithi102_474039_object115:head# 2024-04-25T09:27:40.841 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac8150e5:::benchmark_data_smithi102_474039_object264:head# 2024-04-25T09:27:40.842 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ac8d79cb:::benchmark_data_smithi102_474039_object297:head# 2024-04-25T09:27:40.842 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:afd399f3:::benchmark_data_smithi102_474039_object215:head# 2024-04-25T09:27:40.843 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b19073dd:::benchmark_data_smithi102_474039_object252:head# 2024-04-25T09:27:40.843 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b27390ce:::benchmark_data_smithi102_474039_object181:head# 2024-04-25T09:27:40.844 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:27:40.845 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b9421983:::benchmark_data_smithi102_474039_object139:head# 2024-04-25T09:27:40.846 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:b9d273b5:::benchmark_data_smithi102_474039_object33:head# 2024-04-25T09:27:40.846 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ba160082:::benchmark_data_smithi102_474039_object234:head# 2024-04-25T09:27:40.847 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ba6284dd:::benchmark_data_smithi102_474039_object242:head# 2024-04-25T09:27:40.847 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bc19e3a0:::benchmark_data_smithi102_474039_object209:head# 2024-04-25T09:27:40.848 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bc2a4a6c:::benchmark_data_smithi102_474039_object222:head# 2024-04-25T09:27:40.848 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bc33752a:::benchmark_data_smithi102_474039_object197:head# 2024-04-25T09:27:40.849 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bc3ff702:::benchmark_data_smithi102_474039_object147:head# 2024-04-25T09:27:40.850 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bcc9e0a2:::benchmark_data_smithi102_474039_object25:head# 2024-04-25T09:27:40.850 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:be793a01:::benchmark_data_smithi102_474039_object281:head# 2024-04-25T09:27:40.851 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bee30d45:::benchmark_data_smithi102_474039_object0:head# 2024-04-25T09:27:40.852 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bf44284c:::benchmark_data_smithi102_474039_object179:head# 2024-04-25T09:27:40.852 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bfb81216:::benchmark_data_smithi102_474039_object185:head# 2024-04-25T09:27:40.853 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bfdf06fe:::benchmark_data_smithi102_474039_object77:head# 2024-04-25T09:27:40.853 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:bffabf0a:::benchmark_data_smithi102_474039_object3:head# 2024-04-25T09:27:40.854 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c01db5d2:::benchmark_data_smithi102_474039_object57:head# 2024-04-25T09:27:40.854 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c0700932:::benchmark_data_smithi102_474039_object235:head# 2024-04-25T09:27:40.855 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c1322bc4:::benchmark_data_smithi102_474039_object20:head# 2024-04-25T09:27:40.855 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c30e89b9:::benchmark_data_smithi102_474039_object294:head# 2024-04-25T09:27:40.856 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c3277cfa:::benchmark_data_smithi102_474039_object315:head# 2024-04-25T09:27:40.857 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c3af959e:::benchmark_data_smithi102_474039_object121:head# 2024-04-25T09:27:40.857 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c3f8a0e1:::benchmark_data_smithi102_474039_object224:head# 2024-04-25T09:27:40.858 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c49e7625:::benchmark_data_smithi102_474039_object65:head# 2024-04-25T09:27:40.858 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c4e5cbaf:::benchmark_data_smithi102_474039_object67:head# 2024-04-25T09:27:40.859 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c55165d3:::benchmark_data_smithi102_474039_object214:head# 2024-04-25T09:27:40.859 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c588384f:::benchmark_data_smithi102_474039_object37:head# 2024-04-25T09:27:40.860 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c616460a:::benchmark_data_smithi102_474039_object76:head# 2024-04-25T09:27:40.860 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c68d08c9:::benchmark_data_smithi102_474039_object13:head# 2024-04-25T09:27:40.861 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c87964ee:::benchmark_data_smithi102_474039_object216:head# 2024-04-25T09:27:40.861 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:c918679c:::benchmark_data_smithi102_474039_object48:head# 2024-04-25T09:27:40.862 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ca520e2f:::benchmark_data_smithi102_474039_object254:head# 2024-04-25T09:27:40.863 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ca79d168:::benchmark_data_smithi102_474039_object303:head# 2024-04-25T09:27:40.863 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:caa484d8:::benchmark_data_smithi102_474039_object267:head# 2024-04-25T09:27:40.864 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cacac194:::benchmark_data_smithi102_474039_object225:head# 2024-04-25T09:27:40.865 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cb976fd4:::benchmark_data_smithi102_474039_object187:head# 2024-04-25T09:27:40.865 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ccae141d:::benchmark_data_smithi102_474039_object112:head# 2024-04-25T09:27:40.866 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ce7d3744:::benchmark_data_smithi102_474039_object105:head# 2024-04-25T09:27:40.867 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:cf5251cd:::benchmark_data_smithi102_474039_object173:head# 2024-04-25T09:27:40.867 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d03a4a20:::benchmark_data_smithi102_474039_object273:head# 2024-04-25T09:27:40.868 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d0b37fdd:::benchmark_data_smithi102_474039_object176:head# 2024-04-25T09:27:40.869 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d1525f00:::benchmark_data_smithi102_474039_object182:head# 2024-04-25T09:27:40.869 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d1e2e583:::benchmark_data_smithi102_474039_object134:head# 2024-04-25T09:27:40.870 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d22c4fc4:::benchmark_data_smithi102_474039_object107:head# 2024-04-25T09:27:40.870 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d3676469:::benchmark_data_smithi102_474039_object257:head# 2024-04-25T09:27:40.871 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d54af8ff:::benchmark_data_smithi102_474039_object243:head# 2024-04-25T09:27:40.872 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d55dc631:::benchmark_data_smithi102_474039_object135:head# 2024-04-25T09:27:40.872 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d57afc5e:::benchmark_data_smithi102_474039_object239:head# 2024-04-25T09:27:40.873 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d59a5c6f:::benchmark_data_smithi102_474039_object114:head# 2024-04-25T09:27:40.874 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d5c6e890:::benchmark_data_smithi102_474039_object55:head# 2024-04-25T09:27:40.874 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d6176eb6:::benchmark_data_smithi102_474039_object166:head# 2024-04-25T09:27:40.875 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d7092e79:::benchmark_data_smithi102_474039_object184:head# 2024-04-25T09:27:40.875 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d82cc540:::benchmark_data_smithi102_474039_object228:head# 2024-04-25T09:27:40.876 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d8845e1d:::benchmark_data_smithi102_474039_object1:head# 2024-04-25T09:27:40.876 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:d9bfc5ee:::benchmark_data_smithi102_474039_object206:head# 2024-04-25T09:27:40.877 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:da26776c:::benchmark_data_smithi102_474039_object160:head# 2024-04-25T09:27:40.878 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:db102a11:::benchmark_data_smithi102_474039_object81:head# 2024-04-25T09:27:40.878 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:db5c7cb9:::benchmark_data_smithi102_474039_object150:head# 2024-04-25T09:27:40.879 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dbdbad05:::benchmark_data_smithi102_474039_object101:head# 2024-04-25T09:27:40.879 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dd4d9c69:::benchmark_data_smithi102_474039_object12:head# 2024-04-25T09:27:40.880 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dd4e1b82:::benchmark_data_smithi102_474039_object310:head# 2024-04-25T09:27:40.881 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:df5fb060:::benchmark_data_smithi102_474039_object119:head# 2024-04-25T09:27:40.881 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dfacd7bf:::benchmark_data_smithi102_474039_object70:head# 2024-04-25T09:27:40.882 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:dfd9d5b5:::benchmark_data_smithi102_474039_object174:head# 2024-04-25T09:27:40.883 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e010c870:::benchmark_data_smithi102_474039_object259:head# 2024-04-25T09:27:40.883 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e186b86e:::benchmark_data_smithi102_474039_object309:head# 2024-04-25T09:27:40.884 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e56f5a89:::benchmark_data_smithi102_474039_object152:head# 2024-04-25T09:27:40.884 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e66f5dbd:::benchmark_data_smithi102_474039_object131:head# 2024-04-25T09:27:40.885 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e6bfbf39:::benchmark_data_smithi102_474039_object237:head# 2024-04-25T09:27:40.885 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e6d03ffb:::benchmark_data_smithi102_474039_object64:head# 2024-04-25T09:27:40.886 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e762bddb:::benchmark_data_smithi102_474039_object198:head# 2024-04-25T09:27:40.887 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e7bfdaaf:::benchmark_data_smithi102_474039_object172:head# 2024-04-25T09:27:40.887 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e9757145:::benchmark_data_smithi102_474039_object161:head# 2024-04-25T09:27:40.888 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:e9d173d9:::benchmark_data_smithi102_474039_object308:head# 2024-04-25T09:27:40.889 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ea1aeea3:::benchmark_data_smithi102_474039_object196:head# 2024-04-25T09:27:40.890 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:eaacd3fc:::benchmark_data_smithi102_474039_object261:head# 2024-04-25T09:27:40.890 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:eb6042a7:::benchmark_data_smithi102_474039_object320:head# 2024-04-25T09:27:40.891 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ec593f7d:::benchmark_data_smithi102_474039_object157:head# 2024-04-25T09:27:40.891 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ed72cdc3:::benchmark_data_smithi102_474039_object266:head# 2024-04-25T09:27:40.892 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:edfbd108:::benchmark_data_smithi102_474039_object199:head# 2024-04-25T09:27:40.893 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f033aeef:::benchmark_data_smithi102_474039_object149:head# 2024-04-25T09:27:40.894 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f08b9ea6:::benchmark_data_smithi102_474039_object87:head# 2024-04-25T09:27:40.894 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f09ab9bd:::benchmark_data_smithi102_474039_object61:head# 2024-04-25T09:27:40.895 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f2ba6301:::benchmark_data_smithi102_474039_object30:head# 2024-04-25T09:27:40.895 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f2f6c7dc:::benchmark_data_smithi102_474039_object192:head# 2024-04-25T09:27:40.896 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f3685838:::benchmark_data_smithi102_474039_object21:head# 2024-04-25T09:27:40.896 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f5676a95:::benchmark_data_smithi102_474039_object18:head# 2024-04-25T09:27:40.897 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f5b6f02e:::benchmark_data_smithi102_474039_object47:head# 2024-04-25T09:27:40.897 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f71eddbf:::benchmark_data_smithi102_474039_object304:head# 2024-04-25T09:27:40.898 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f7fc3b27:::benchmark_data_smithi102_474039_object120:head# 2024-04-25T09:27:40.898 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:f8518d9b:::benchmark_data_smithi102_474039_object44:head# 2024-04-25T09:27:40.899 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fab39e4b:::benchmark_data_smithi102_474039_object106:head# 2024-04-25T09:27:40.899 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fb1138dd:::benchmark_data_smithi102_474039_object240:head# 2024-04-25T09:27:40.900 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fb96aee6:::benchmark_data_smithi102_474039_object245:head# 2024-04-25T09:27:40.900 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fc30d971:::benchmark_data_smithi102_474039_object58:head# 2024-04-25T09:27:40.901 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fd9e2883:::benchmark_data_smithi102_474039_object38:head# 2024-04-25T09:27:40.901 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fe977974:::benchmark_data_smithi102_474039_object286:head# 2024-04-25T09:27:40.902 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:fede8008:::benchmark_data_smithi102_474039_object62:head# 2024-04-25T09:27:40.903 INFO:tasks.workunit.client.0.smithi102.stderr:Read #1:ff347bd7:::benchmark_data_smithi102_474039_object129:head# 2024-04-25T09:27:40.903 INFO:tasks.workunit.client.0.smithi102.stderr:Export successful 2024-04-25T09:27:41.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:175: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-04-25T09:27:41.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:27:41.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:27:41.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:27:41.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:41.191 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:27:41.192 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:27:41.193 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:27:41.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:27:41.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:27:41.194 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:27:41.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:27:41.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:27:41.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:27:41.195 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:27:41.229 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:27:41.243+0000 7f3da8a3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:41.231 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:27:41.247+0000 7f3da8a3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:41.233 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:27:41.247+0000 7f3da8a3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:27:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:27:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:27:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:27:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:27:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:41.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:27:41.631 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:27:41.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:41.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:41.791 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:27:41.807+0000 7f3da8a3a7c0 -1 Falling back to public interface 2024-04-25T09:27:42.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:43.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:43.056 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:43.056 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:27:43.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:27:43.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:43.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:43.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:44.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:44.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:44.490 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:27:44.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:27:44.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:44.491 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:44.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:45.005 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:27:45.019+0000 7f3da8a3a7c0 -1 osd.0 9 log_to_monitors true 2024-04-25T09:27:45.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:45.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:45.986 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:27:45.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:27:45.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:45.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:46.426 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 12 up_thru 12 down_at 10 last_clean_interval [5,9) [v2:127.0.0.1:6802/1165113832,v1:127.0.0.1:6803/1165113832] [v2:127.0.0.1:6804/1165113832,v1:127.0.0.1:6805/1165113832] exists,up b7b813c4-be34-4c65-abf1-0e52eb14287b 2024-04-25T09:27:46.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:27:46.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:27:46.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:27:46.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:177: TEST_import_after_split: ceph osd pool set foo pg_num 2 2024-04-25T09:27:47.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:178: TEST_import_after_split: sleep 5 2024-04-25T09:27:52.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: ceph daemon osd.0 perf dump 2024-04-25T09:27:52.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: jq .osd.numpg 2024-04-25T09:27:52.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: grep 1 2024-04-25T09:27:52.383 INFO:tasks.workunit.client.0.smithi102.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-25T09:27:52.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:180: TEST_import_after_split: wait_for_clean 2024-04-25T09:27:52.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:27:52.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:27:52.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:27:52.386 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:27:52.386 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:27:52.386 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:27:52.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:27:52.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:27:52.387 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:27:52.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:27:52.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:27:52.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:27:52.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:27:52.492 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:27:52.492 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:27:52.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:27:52.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:27:52.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:27:52.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:27:53.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607556 2024-04-25T09:27:53.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607556 2024-04-25T09:27:53.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-51539607556' 2024-04-25T09:27:53.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:27:53.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-51539607556 2024-04-25T09:27:53.153 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:27:53.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:27:53.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-51539607556 2024-04-25T09:27:53.155 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:27:53.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607556 2024-04-25T09:27:53.156 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 51539607556 2024-04-25T09:27:53.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 51539607556' 2024-04-25T09:27:53.157 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:27:53.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607554 -lt 51539607556 2024-04-25T09:27:53.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:27:54.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:27:54.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:27:54.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607556 -lt 51539607556 2024-04-25T09:27:54.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:27:54.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:27:54.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:27:55.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:27:55.569 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:27:55.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:27:55.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:27:55.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:27:55.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:27:55.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:27:55.569 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:27:55.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:27:55.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:27:55.978 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:27:55.979 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:27:56.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:27:56.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:27:56.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:27:56.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:182: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:27:56.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:27:56.521 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:27:56.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:27:56.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:27:56.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:27:56.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:27:56.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:184: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:27:58.198 INFO:tasks.workunit.client.0.smithi102.stdout: marking collection for removal 2024-04-25T09:27:58.198 INFO:tasks.workunit.client.0.smithi102.stdout:setting '_remove' omap key 2024-04-25T09:27:58.198 INFO:tasks.workunit.client.0.smithi102.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:27:58.531 INFO:tasks.workunit.client.0.smithi102.stdout:Remove successful 2024-04-25T09:27:58.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:187: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:28:00.136 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-04-25T09:28:00.137 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.0 2024-04-25T09:28:00.137 INFO:tasks.workunit.client.0.smithi102.stdout:pool 1 pg_num 1 -> 2 2024-04-25T09:28:00.138 INFO:tasks.workunit.client.0.smithi102.stderr: children are 1.1 2024-04-25T09:28:00.138 INFO:tasks.workunit.client.0.smithi102.stderr:pgid 1.0 splits to 1.1 and 1.1 exists 2024-04-25T09:28:00.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:189: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-04-25T09:28:01.040 INFO:tasks.workunit.client.0.smithi102.stdout: marking collection for removal 2024-04-25T09:28:01.040 INFO:tasks.workunit.client.0.smithi102.stdout:setting '_remove' omap key 2024-04-25T09:28:01.040 INFO:tasks.workunit.client.0.smithi102.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:28:01.313 INFO:tasks.workunit.client.0.smithi102.stdout:Remove successful 2024-04-25T09:28:01.838 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:191: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:28:02.833 INFO:tasks.workunit.client.0.smithi102.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-04-25T09:28:02.833 INFO:tasks.workunit.client.0.smithi102.stdout:Importing pgid 1.0 2024-04-25T09:28:02.833 INFO:tasks.workunit.client.0.smithi102.stdout:pool 1 pg_num 1 -> 2 2024-04-25T09:28:02.834 INFO:tasks.workunit.client.0.smithi102.stderr: children are 1.1 2024-04-25T09:28:02.834 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:00100e1f:::benchmark_data_smithi102_474039_object285:head# 2024-04-25T09:28:02.834 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:02.957 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:016d36bd:::benchmark_data_smithi102_474039_object248:head# 2024-04-25T09:28:02.958 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:02.999 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:02ab28e4:::benchmark_data_smithi102_474039_object287:head# 2024-04-25T09:28:02.999 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.041 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:02fc2004:::benchmark_data_smithi102_474039_object307:head# 2024-04-25T09:28:03.041 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.083 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:030f3311:::benchmark_data_smithi102_474039_object306:head# 2024-04-25T09:28:03.083 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.125 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:034cce74:::benchmark_data_smithi102_474039_object207:head# 2024-04-25T09:28:03.125 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.175 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:037b8264:::benchmark_data_smithi102_474039_object19:head# 2024-04-25T09:28:03.175 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.217 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0473b718:::benchmark_data_smithi102_474039_object220:head# 2024-04-25T09:28:03.217 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.259 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0508c379:::benchmark_data_smithi102_474039_object162:head# 2024-04-25T09:28:03.259 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.300 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:064ec71f:::benchmark_data_smithi102_474039_object71:head# 2024-04-25T09:28:03.300 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.342 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:06f42a1b:::benchmark_data_smithi102_474039_object263:head# 2024-04-25T09:28:03.342 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.384 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:07db957e:::benchmark_data_smithi102_474039_object98:head# 2024-04-25T09:28:03.384 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.426 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:07e0f0ab:::benchmark_data_smithi102_474039_object291:head# 2024-04-25T09:28:03.426 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.467 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:087fa2a2:::benchmark_data_smithi102_474039_object91:head# 2024-04-25T09:28:03.468 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.509 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:091e887c:::benchmark_data_smithi102_474039_object53:head# 2024-04-25T09:28:03.510 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.551 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0a42931d:::benchmark_data_smithi102_474039_object301:head# 2024-04-25T09:28:03.551 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.593 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0a4a118b:::benchmark_data_smithi102_474039_object223:head# 2024-04-25T09:28:03.593 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.635 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0af76ace:::benchmark_data_smithi102_474039_object262:head# 2024-04-25T09:28:03.635 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.677 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0b71271b:::benchmark_data_smithi102_474039_object312:head# 2024-04-25T09:28:03.677 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.718 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0baa67d4:::benchmark_data_smithi102_474039_object171:head# 2024-04-25T09:28:03.719 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.760 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0cd0abf2:::benchmark_data_smithi102_474039_object14:head# 2024-04-25T09:28:03.760 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.802 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0d156d14:::benchmark_data_smithi102_474039_object313:head# 2024-04-25T09:28:03.802 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.844 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0dfdd378:::benchmark_data_smithi102_474039_object295:head# 2024-04-25T09:28:03.844 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.885 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0ec0b5ce:::benchmark_data_smithi102_474039_object204:head# 2024-04-25T09:28:03.885 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.927 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0ed0639c:::benchmark_data_smithi102_474039_object136:head# 2024-04-25T09:28:03.927 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:03.969 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0f5eb555:::benchmark_data_smithi102_474039_object80:head# 2024-04-25T09:28:03.969 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.011 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0f9cd47b:::benchmark_data_smithi102_474039_object232:head# 2024-04-25T09:28:04.011 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.044 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:0fad4256:::benchmark_data_smithi102_474039_object113:head# 2024-04-25T09:28:04.044 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.078 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:119d5994:::benchmark_data_smithi102_474039_object280:head# 2024-04-25T09:28:04.078 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.111 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:12683126:::benchmark_data_smithi102_474039_object82:head# 2024-04-25T09:28:04.111 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.186 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:12b6f85b:::benchmark_data_smithi102_474039_object151:head# 2024-04-25T09:28:04.187 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.220 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:15002f77:::benchmark_data_smithi102_474039_object34:head# 2024-04-25T09:28:04.220 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.253 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:168415d3:::benchmark_data_smithi102_474039_object5:head# 2024-04-25T09:28:04.253 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.287 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:16baadaa:::benchmark_data_smithi102_474039_object175:head# 2024-04-25T09:28:04.287 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.320 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:18143727:::benchmark_data_smithi102_474039_object128:head# 2024-04-25T09:28:04.320 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.354 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1845f65a:::benchmark_data_smithi102_474039_object217:head# 2024-04-25T09:28:04.354 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.387 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1858dfac:::benchmark_data_smithi102_474039_object269:head# 2024-04-25T09:28:04.387 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.421 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:18708f85:::benchmark_data_smithi102_474039_object278:head# 2024-04-25T09:28:04.421 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.454 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:19d44365:::benchmark_data_smithi102_474039_object56:head# 2024-04-25T09:28:04.455 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.487 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:19eb82fa:::benchmark_data_smithi102_474039_object108:head# 2024-04-25T09:28:04.488 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.521 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1a051d39:::benchmark_data_smithi102_474039_object311:head# 2024-04-25T09:28:04.521 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.554 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1a9350f4:::benchmark_data_smithi102_474039_object86:head# 2024-04-25T09:28:04.554 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.588 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1a9d0683:::benchmark_data_smithi102_474039_object63:head# 2024-04-25T09:28:04.588 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.622 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1b8bb510:::benchmark_data_smithi102_474039_object302:head# 2024-04-25T09:28:04.622 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.655 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1bfa1058:::benchmark_data_smithi102_474039_object117:head# 2024-04-25T09:28:04.655 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.722 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1e1f8f1c:::benchmark_data_smithi102_474039_object203:head# 2024-04-25T09:28:04.722 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.789 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:1f50317b:::benchmark_data_smithi102_474039_object42:head# 2024-04-25T09:28:04.789 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.864 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:20c4c6a3:::benchmark_data_smithi102_474039_object95:head# 2024-04-25T09:28:04.864 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.906 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:218a2cb2:::benchmark_data_smithi102_474039_object282:head# 2024-04-25T09:28:04.906 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:04.981 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:242d6fe2:::benchmark_data_smithi102_474039_object158:head# 2024-04-25T09:28:04.981 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.023 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:25e1ed0c:::benchmark_data_smithi102_474039_object137:head# 2024-04-25T09:28:05.023 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.065 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:272b798a:::benchmark_data_smithi102_474039_object100:head# 2024-04-25T09:28:05.065 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.107 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:283127ae:::benchmark_data_smithi102_474039_object193:head# 2024-04-25T09:28:05.107 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.148 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:288243f9:::benchmark_data_smithi102_474039_object153:head# 2024-04-25T09:28:05.149 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.190 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:28cc675e:::benchmark_data_smithi102_474039_object318:head# 2024-04-25T09:28:05.191 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.232 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2c04df7f:::benchmark_data_smithi102_474039_object79:head# 2024-04-25T09:28:05.232 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.274 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:2dda7f96:::benchmark_data_smithi102_474039_object144:head# 2024-04-25T09:28:05.274 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.315 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:31df5474:::benchmark_data_smithi102_474039_object283:head# 2024-04-25T09:28:05.316 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.357 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:32997b14:::benchmark_data_smithi102_474039_object22:head# 2024-04-25T09:28:05.357 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.399 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3362a17e:::benchmark_data_smithi102_474039_object74:head# 2024-04-25T09:28:05.399 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.441 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:356c603e:::benchmark_data_smithi102_474039_object221:head# 2024-04-25T09:28:05.441 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.483 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:366c2233:::benchmark_data_smithi102_474039_object10:head# 2024-04-25T09:28:05.483 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.524 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:36d21c33:::benchmark_data_smithi102_474039_object205:head# 2024-04-25T09:28:05.525 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.575 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:37084488:::benchmark_data_smithi102_474039_object190:head# 2024-04-25T09:28:05.575 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.633 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3713e5dc:::benchmark_data_smithi102_474039_object69:head# 2024-04-25T09:28:05.634 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.683 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:389d97cb:::benchmark_data_smithi102_474039_object274:head# 2024-04-25T09:28:05.683 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.725 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3a5a88b7:::benchmark_data_smithi102_474039_object26:head# 2024-04-25T09:28:05.725 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.767 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3b32d00d:::benchmark_data_smithi102_474039_object219:head# 2024-04-25T09:28:05.767 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.809 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3c35bebb:::benchmark_data_smithi102_474039_object159:head# 2024-04-25T09:28:05.809 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.851 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3c7f614f:::benchmark_data_smithi102_474039_object7:head# 2024-04-25T09:28:05.851 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.892 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3d2b26a7:::benchmark_data_smithi102_474039_object17:head# 2024-04-25T09:28:05.892 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.934 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3d471739:::benchmark_data_smithi102_474039_object97:head# 2024-04-25T09:28:05.934 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:05.976 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3e0c45d9:::benchmark_data_smithi102_474039_object51:head# 2024-04-25T09:28:05.976 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.018 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:3e52bb25:::benchmark_data_smithi102_474039_object258:head# 2024-04-25T09:28:06.018 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.059 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:412fb915:::benchmark_data_smithi102_474039_object156:head# 2024-04-25T09:28:06.060 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.101 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4170f76c:::benchmark_data_smithi102_474039_object143:head# 2024-04-25T09:28:06.101 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.168 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4284d7d2:::benchmark_data_smithi102_474039_object11:head# 2024-04-25T09:28:06.168 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.227 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4362be4d:::benchmark_data_smithi102_474039_object145:head# 2024-04-25T09:28:06.227 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.269 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:43989524:::benchmark_data_smithi102_474039_object16:head# 2024-04-25T09:28:06.269 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.310 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:455fdb9c:::benchmark_data_smithi102_474039_object88:head# 2024-04-25T09:28:06.311 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.352 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:47245629:::benchmark_data_smithi102_474039_object66:head# 2024-04-25T09:28:06.352 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.394 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4782c34d:::benchmark_data_smithi102_474039_object180:head# 2024-04-25T09:28:06.394 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.436 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:493b10f5:::benchmark_data_smithi102_474039_object236:head# 2024-04-25T09:28:06.436 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.477 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:49a2c41e:::benchmark_data_smithi102_474039_object125:head# 2024-04-25T09:28:06.478 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.519 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:49bfbf8f:::benchmark_data_smithi102_474039_object111:head# 2024-04-25T09:28:06.519 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.561 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:49d2acb0:::benchmark_data_smithi102_474039_object84:head# 2024-04-25T09:28:06.561 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.603 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4ada50a6:::benchmark_data_smithi102_474039_object109:head# 2024-04-25T09:28:06.603 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.645 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4b4086e5:::benchmark_data_smithi102_474039_object275:head# 2024-04-25T09:28:06.645 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.701 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4d2b067d:::benchmark_data_smithi102_474039_object256:head# 2024-04-25T09:28:06.701 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.787 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4e4a078b:::benchmark_data_smithi102_474039_object102:head# 2024-04-25T09:28:06.787 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.854 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4e98afd2:::benchmark_data_smithi102_474039_object229:head# 2024-04-25T09:28:06.854 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.888 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4ebffa2f:::benchmark_data_smithi102_474039_object251:head# 2024-04-25T09:28:06.888 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.921 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4ec71e53:::benchmark_data_smithi102_474039_object183:head# 2024-04-25T09:28:06.921 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.954 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:4ff94952:::benchmark_data_smithi102_474039_object6:head# 2024-04-25T09:28:06.955 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:06.988 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5121d7ac:::benchmark_data_smithi102_474039_object90:head# 2024-04-25T09:28:06.988 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.021 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:51a8b164:::benchmark_data_smithi102_474039_object233:head# 2024-04-25T09:28:07.021 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.055 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5303b8d7:::benchmark_data_smithi102_474039_object4:head# 2024-04-25T09:28:07.055 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.088 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:560fc05e:::benchmark_data_smithi102_474039_object246:head# 2024-04-25T09:28:07.089 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.122 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:58166439:::benchmark_data_smithi102_474039_object32:head# 2024-04-25T09:28:07.122 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.155 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5881a447:::benchmark_data_smithi102_474039_object35:head# 2024-04-25T09:28:07.155 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.189 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:58f91e48:::benchmark_data_smithi102_474039_object103:head# 2024-04-25T09:28:07.189 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.222 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:59ab0008:::benchmark_data_smithi102_474039_object68:head# 2024-04-25T09:28:07.222 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.255 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:59b4e707:::benchmark_data_smithi102_474039_object284:head# 2024-04-25T09:28:07.256 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.289 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:59b7abdf:::benchmark_data_smithi102_474039_object186:head# 2024-04-25T09:28:07.289 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.322 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5b0b0b70:::benchmark_data_smithi102_474039_object168:head# 2024-04-25T09:28:07.323 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.356 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5cc9558d:::benchmark_data_smithi102_474039_object201:head# 2024-04-25T09:28:07.356 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.389 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5cf45479:::benchmark_data_smithi102_474039_object316:head# 2024-04-25T09:28:07.389 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.423 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5d7d41eb:::benchmark_data_smithi102_474039_object202:head# 2024-04-25T09:28:07.423 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.456 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5d8c345c:::benchmark_data_smithi102_474039_object317:head# 2024-04-25T09:28:07.456 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.490 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5dca7eef:::benchmark_data_smithi102_474039_object195:head# 2024-04-25T09:28:07.490 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.532 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5e27b06c:::benchmark_data_smithi102_474039_object154:head# 2024-04-25T09:28:07.532 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.573 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:5f003aea:::benchmark_data_smithi102_474039_object122:head# 2024-04-25T09:28:07.574 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.615 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:60982b13:::benchmark_data_smithi102_474039_object27:head# 2024-04-25T09:28:07.615 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.657 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:610cfa39:::benchmark_data_smithi102_474039_object118:head# 2024-04-25T09:28:07.657 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.699 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:617982b8:::benchmark_data_smithi102_474039_object277:head# 2024-04-25T09:28:07.699 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.799 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:61fe964f:::benchmark_data_smithi102_474039_object210:head# 2024-04-25T09:28:07.799 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.841 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6235c3e6:::benchmark_data_smithi102_474039_object189:head# 2024-04-25T09:28:07.841 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.882 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:638377f6:::benchmark_data_smithi102_474039_object177:head# 2024-04-25T09:28:07.883 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.924 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:638df456:::benchmark_data_smithi102_474039_object104:head# 2024-04-25T09:28:07.924 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:07.966 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:64248811:::benchmark_data_smithi102_474039_object155:head# 2024-04-25T09:28:07.966 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.066 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:677c0aaa:::benchmark_data_smithi102_474039_object227:head# 2024-04-25T09:28:08.066 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.108 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:67a2c881:::benchmark_data_smithi102_474039_object39:head# 2024-04-25T09:28:08.108 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.175 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:67ebb9ef:::benchmark_data_smithi102_474039_object148:head# 2024-04-25T09:28:08.175 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.250 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:68764934:::benchmark_data_smithi102_474039_object255:head# 2024-04-25T09:28:08.251 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.317 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6944e27b:::benchmark_data_smithi102_474039_object60:head# 2024-04-25T09:28:08.318 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.401 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:69b1a42e:::benchmark_data_smithi102_474039_object218:head# 2024-04-25T09:28:08.401 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.493 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6a07e9f8:::benchmark_data_smithi102_474039_object28:head# 2024-04-25T09:28:08.493 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.535 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6a080ceb:::benchmark_data_smithi102_474039_object40:head# 2024-04-25T09:28:08.535 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.577 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6a418838:::benchmark_data_smithi102_474039_object250:head# 2024-04-25T09:28:08.577 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.627 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6a91d9dc:::benchmark_data_smithi102_474039_object208:head# 2024-04-25T09:28:08.627 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.682 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6afabd1c:::benchmark_data_smithi102_474039_object140:head# 2024-04-25T09:28:08.682 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.761 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6c5b143e:::benchmark_data_smithi102_474039_object270:head# 2024-04-25T09:28:08.761 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.828 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6c82857d:::benchmark_data_smithi102_474039_object226:head# 2024-04-25T09:28:08.828 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.886 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6cce6c95:::benchmark_data_smithi102_474039_object49:head# 2024-04-25T09:28:08.887 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.928 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:6ea142d3:::benchmark_data_smithi102_474039_object41:head# 2024-04-25T09:28:08.928 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:08.970 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7023b0bd:::benchmark_data_smithi102_474039_object124:head# 2024-04-25T09:28:08.970 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.012 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:706481cb:::benchmark_data_smithi102_474039_object298:head# 2024-04-25T09:28:09.012 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.053 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:716b84ca:::benchmark_data_smithi102_474039_object165:head# 2024-04-25T09:28:09.054 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.095 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:71dad963:::benchmark_data_smithi102_474039_object130:head# 2024-04-25T09:28:09.095 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.137 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7298df79:::benchmark_data_smithi102_474039_object72:head# 2024-04-25T09:28:09.137 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.179 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:74627db4:::benchmark_data_smithi102_474039_object238:head# 2024-04-25T09:28:09.179 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.221 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:747bef36:::benchmark_data_smithi102_474039_object146:head# 2024-04-25T09:28:09.221 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.262 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:74c700d3:::benchmark_data_smithi102_474039_object200:head# 2024-04-25T09:28:09.263 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.304 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:756f95d3:::benchmark_data_smithi102_474039_object127:head# 2024-04-25T09:28:09.304 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.338 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7619e673:::benchmark_data_smithi102_474039_object94:head# 2024-04-25T09:28:09.338 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.371 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:76449db9:::benchmark_data_smithi102_474039_object141:head# 2024-04-25T09:28:09.371 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.405 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7867171d:::benchmark_data_smithi102_474039_object92:head# 2024-04-25T09:28:09.405 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.438 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:791732fc:::benchmark_data_smithi102_474039_object89:head# 2024-04-25T09:28:09.438 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.471 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:792876a9:::benchmark_data_smithi102_474039_object271:head# 2024-04-25T09:28:09.472 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.505 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7a8c047a:::benchmark_data_smithi102_474039_object126:head# 2024-04-25T09:28:09.505 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.538 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7b184e43:::benchmark_data_smithi102_474039_object52:head# 2024-04-25T09:28:09.539 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.572 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7bddf6f7:::benchmark_data_smithi102_474039_object132:head# 2024-04-25T09:28:09.572 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.605 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:7df96f0f:::benchmark_data_smithi102_474039_object15:head# 2024-04-25T09:28:09.605 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.639 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:80328a68:::benchmark_data_smithi102_474039_object247:head# 2024-04-25T09:28:09.639 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.672 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:80668dd0:::benchmark_data_smithi102_474039_object169:head# 2024-04-25T09:28:09.672 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.706 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:80c889ae:::benchmark_data_smithi102_474039_object43:head# 2024-04-25T09:28:09.706 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.739 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8278fb4f:::benchmark_data_smithi102_474039_object289:head# 2024-04-25T09:28:09.739 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.773 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:831dc679:::benchmark_data_smithi102_474039_object36:head# 2024-04-25T09:28:09.830 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.831 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:84902432:::benchmark_data_smithi102_474039_object24:head# 2024-04-25T09:28:09.831 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.839 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:86592b4a:::benchmark_data_smithi102_474039_object260:head# 2024-04-25T09:28:09.839 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.873 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:87a9131c:::benchmark_data_smithi102_474039_object244:head# 2024-04-25T09:28:09.873 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.906 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:87e3ba06:::benchmark_data_smithi102_474039_object268:head# 2024-04-25T09:28:09.907 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.940 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:88267fa3:::benchmark_data_smithi102_474039_object116:head# 2024-04-25T09:28:09.940 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:09.973 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:88706ab9:::benchmark_data_smithi102_474039_object59:head# 2024-04-25T09:28:09.973 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.007 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:88a98f2c:::benchmark_data_smithi102_474039_object8:head# 2024-04-25T09:28:10.007 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.040 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:899ddbe7:::benchmark_data_smithi102_474039_object170:head# 2024-04-25T09:28:10.040 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.074 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8c31299f:::benchmark_data_smithi102_474039_object142:head# 2024-04-25T09:28:10.074 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.107 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8c659197:::benchmark_data_smithi102_474039_object305:head# 2024-04-25T09:28:10.107 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.174 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8d5bcc76:::benchmark_data_smithi102_474039_object164:head# 2024-04-25T09:28:10.174 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.224 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8de977a4:::benchmark_data_smithi102_474039_object276:head# 2024-04-25T09:28:10.225 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.266 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8e54a6f0:::benchmark_data_smithi102_474039_object123:head# 2024-04-25T09:28:10.267 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.308 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8eb9693c:::benchmark_data_smithi102_474039_object299:head# 2024-04-25T09:28:10.308 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.350 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8f268b7b:::benchmark_data_smithi102_474039_object163:head# 2024-04-25T09:28:10.350 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.392 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:8fb5dd81:::benchmark_data_smithi102_474039_object31:head# 2024-04-25T09:28:10.392 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.433 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9015d089:::benchmark_data_smithi102_474039_object292:head# 2024-04-25T09:28:10.434 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.475 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9017a08a:::benchmark_data_smithi102_474039_object212:head# 2024-04-25T09:28:10.475 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.517 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:90930b9a:::benchmark_data_smithi102_474039_object46:head# 2024-04-25T09:28:10.517 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.559 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:91c76f76:::benchmark_data_smithi102_474039_object188:head# 2024-04-25T09:28:10.560 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.626 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9250853e:::benchmark_data_smithi102_474039_object265:head# 2024-04-25T09:28:10.626 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.682 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:92914a1f:::benchmark_data_smithi102_474039_object231:head# 2024-04-25T09:28:10.682 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.751 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:934cbb72:::benchmark_data_smithi102_474039_object75:head# 2024-04-25T09:28:10.751 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.818 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:947c7936:::benchmark_data_smithi102_474039_object110:head# 2024-04-25T09:28:10.818 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.877 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9485f46d:::benchmark_data_smithi102_474039_object290:head# 2024-04-25T09:28:10.877 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.919 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:94b492cf:::benchmark_data_smithi102_474039_object272:head# 2024-04-25T09:28:10.919 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:10.960 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9549b256:::benchmark_data_smithi102_474039_object9:head# 2024-04-25T09:28:10.961 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.002 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:95e884e1:::benchmark_data_smithi102_474039_object249:head# 2024-04-25T09:28:11.002 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.044 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9621544e:::benchmark_data_smithi102_474039_object279:head# 2024-04-25T09:28:11.044 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.111 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:965f10ca:::benchmark_data_smithi102_474039_object83:head# 2024-04-25T09:28:11.111 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.154 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:97115da6:::benchmark_data_smithi102_474039_object85:head# 2024-04-25T09:28:11.154 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.194 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:97e3beb2:::benchmark_data_smithi102_474039_object213:head# 2024-04-25T09:28:11.195 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.236 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:98f17a70:::benchmark_data_smithi102_474039_object45:head# 2024-04-25T09:28:11.236 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.278 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:99bb20a2:::benchmark_data_smithi102_474039_object73:head# 2024-04-25T09:28:11.278 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.320 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:99c72720:::benchmark_data_smithi102_474039_object211:head# 2024-04-25T09:28:11.320 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.403 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9a3605e2:::benchmark_data_smithi102_474039_object230:head# 2024-04-25T09:28:11.403 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.445 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9a896a36:::benchmark_data_smithi102_474039_object93:head# 2024-04-25T09:28:11.445 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.487 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9be82daf:::benchmark_data_smithi102_474039_object191:head# 2024-04-25T09:28:11.487 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.528 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9ceb25b1:::benchmark_data_smithi102_474039_object253:head# 2024-04-25T09:28:11.528 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.570 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9d1ff8ff:::benchmark_data_smithi102_474039_object194:head# 2024-04-25T09:28:11.570 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.612 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9d4981d9:::benchmark_data_smithi102_474039_object167:head# 2024-04-25T09:28:11.612 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.654 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9dbd761d:::benchmark_data_smithi102_474039_object54:head# 2024-04-25T09:28:11.654 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.696 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9dd01cce:::benchmark_data_smithi102_474039_object178:head# 2024-04-25T09:28:11.696 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.737 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9df2ece6:::benchmark_data_smithi102_474039_object78:head# 2024-04-25T09:28:11.738 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.779 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:9ed71fdd:::benchmark_data_smithi102_474039_object29:head# 2024-04-25T09:28:11.779 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.821 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a0621afc:::benchmark_data_smithi102_474039_object133:head# 2024-04-25T09:28:11.821 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.871 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a11c251c:::benchmark_data_smithi102_474039_object293:head# 2024-04-25T09:28:11.871 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.913 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a193ff46:::benchmark_data_smithi102_474039_object314:head# 2024-04-25T09:28:11.913 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.955 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a1d05115:::benchmark_data_smithi102_474039_object296:head# 2024-04-25T09:28:11.955 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:11.997 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a1e17490:::benchmark_data_smithi102_474039_object2:head# 2024-04-25T09:28:11.997 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.038 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a2467ccf:::benchmark_data_smithi102_474039_object23:head# 2024-04-25T09:28:12.039 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.080 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a3caa411:::benchmark_data_smithi102_474039_object99:head# 2024-04-25T09:28:12.080 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.131 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a3d45b93:::benchmark_data_smithi102_474039_object96:head# 2024-04-25T09:28:12.132 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.197 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a4ccd216:::benchmark_data_smithi102_474039_object319:head# 2024-04-25T09:28:12.198 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.247 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a70e57d8:::benchmark_data_smithi102_474039_object300:head# 2024-04-25T09:28:12.247 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.281 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a83cc239:::benchmark_data_smithi102_474039_object138:head# 2024-04-25T09:28:12.281 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.314 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a88493d3:::benchmark_data_smithi102_474039_object50:head# 2024-04-25T09:28:12.314 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.348 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:a9fa51a1:::benchmark_data_smithi102_474039_object241:head# 2024-04-25T09:28:12.348 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.381 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac1cc923:::benchmark_data_smithi102_474039_object288:head# 2024-04-25T09:28:12.381 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.414 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac6944ff:::benchmark_data_smithi102_474039_object115:head# 2024-04-25T09:28:12.415 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.448 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac8150e5:::benchmark_data_smithi102_474039_object264:head# 2024-04-25T09:28:12.448 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.481 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ac8d79cb:::benchmark_data_smithi102_474039_object297:head# 2024-04-25T09:28:12.481 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.515 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:afd399f3:::benchmark_data_smithi102_474039_object215:head# 2024-04-25T09:28:12.515 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.549 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b19073dd:::benchmark_data_smithi102_474039_object252:head# 2024-04-25T09:28:12.549 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.582 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b27390ce:::benchmark_data_smithi102_474039_object181:head# 2024-04-25T09:28:12.582 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.615 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:28:12.616 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.649 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b9421983:::benchmark_data_smithi102_474039_object139:head# 2024-04-25T09:28:12.649 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.716 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:b9d273b5:::benchmark_data_smithi102_474039_object33:head# 2024-04-25T09:28:12.716 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.776 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ba160082:::benchmark_data_smithi102_474039_object234:head# 2024-04-25T09:28:12.776 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.852 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ba6284dd:::benchmark_data_smithi102_474039_object242:head# 2024-04-25T09:28:12.852 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.894 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc19e3a0:::benchmark_data_smithi102_474039_object209:head# 2024-04-25T09:28:12.894 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.936 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc2a4a6c:::benchmark_data_smithi102_474039_object222:head# 2024-04-25T09:28:12.936 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:12.977 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc33752a:::benchmark_data_smithi102_474039_object197:head# 2024-04-25T09:28:12.978 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.019 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bc3ff702:::benchmark_data_smithi102_474039_object147:head# 2024-04-25T09:28:13.019 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.061 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bcc9e0a2:::benchmark_data_smithi102_474039_object25:head# 2024-04-25T09:28:13.061 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.103 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:be793a01:::benchmark_data_smithi102_474039_object281:head# 2024-04-25T09:28:13.103 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.144 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bee30d45:::benchmark_data_smithi102_474039_object0:head# 2024-04-25T09:28:13.145 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.186 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bf44284c:::benchmark_data_smithi102_474039_object179:head# 2024-04-25T09:28:13.187 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.228 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bfb81216:::benchmark_data_smithi102_474039_object185:head# 2024-04-25T09:28:13.228 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.270 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bfdf06fe:::benchmark_data_smithi102_474039_object77:head# 2024-04-25T09:28:13.270 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.312 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:bffabf0a:::benchmark_data_smithi102_474039_object3:head# 2024-04-25T09:28:13.312 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.354 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c01db5d2:::benchmark_data_smithi102_474039_object57:head# 2024-04-25T09:28:13.354 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.395 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c0700932:::benchmark_data_smithi102_474039_object235:head# 2024-04-25T09:28:13.396 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.437 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c1322bc4:::benchmark_data_smithi102_474039_object20:head# 2024-04-25T09:28:13.437 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.479 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c30e89b9:::benchmark_data_smithi102_474039_object294:head# 2024-04-25T09:28:13.479 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.521 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c3277cfa:::benchmark_data_smithi102_474039_object315:head# 2024-04-25T09:28:13.521 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.563 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c3af959e:::benchmark_data_smithi102_474039_object121:head# 2024-04-25T09:28:13.563 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.604 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c3f8a0e1:::benchmark_data_smithi102_474039_object224:head# 2024-04-25T09:28:13.605 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.646 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c49e7625:::benchmark_data_smithi102_474039_object65:head# 2024-04-25T09:28:13.646 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.688 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c4e5cbaf:::benchmark_data_smithi102_474039_object67:head# 2024-04-25T09:28:13.688 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.730 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c55165d3:::benchmark_data_smithi102_474039_object214:head# 2024-04-25T09:28:13.730 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.772 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c588384f:::benchmark_data_smithi102_474039_object37:head# 2024-04-25T09:28:13.772 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.813 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c616460a:::benchmark_data_smithi102_474039_object76:head# 2024-04-25T09:28:13.814 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.855 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c68d08c9:::benchmark_data_smithi102_474039_object13:head# 2024-04-25T09:28:13.855 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.897 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c87964ee:::benchmark_data_smithi102_474039_object216:head# 2024-04-25T09:28:13.897 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.938 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:c918679c:::benchmark_data_smithi102_474039_object48:head# 2024-04-25T09:28:13.939 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:13.980 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ca520e2f:::benchmark_data_smithi102_474039_object254:head# 2024-04-25T09:28:13.981 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.022 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ca79d168:::benchmark_data_smithi102_474039_object303:head# 2024-04-25T09:28:14.023 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.064 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:caa484d8:::benchmark_data_smithi102_474039_object267:head# 2024-04-25T09:28:14.064 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.206 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cacac194:::benchmark_data_smithi102_474039_object225:head# 2024-04-25T09:28:14.206 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.257 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cb976fd4:::benchmark_data_smithi102_474039_object187:head# 2024-04-25T09:28:14.257 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.290 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ccae141d:::benchmark_data_smithi102_474039_object112:head# 2024-04-25T09:28:14.290 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.323 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ce7d3744:::benchmark_data_smithi102_474039_object105:head# 2024-04-25T09:28:14.323 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.357 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:cf5251cd:::benchmark_data_smithi102_474039_object173:head# 2024-04-25T09:28:14.357 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.390 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d03a4a20:::benchmark_data_smithi102_474039_object273:head# 2024-04-25T09:28:14.390 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.427 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d0b37fdd:::benchmark_data_smithi102_474039_object176:head# 2024-04-25T09:28:14.427 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.457 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d1525f00:::benchmark_data_smithi102_474039_object182:head# 2024-04-25T09:28:14.457 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.490 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d1e2e583:::benchmark_data_smithi102_474039_object134:head# 2024-04-25T09:28:14.491 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.524 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d22c4fc4:::benchmark_data_smithi102_474039_object107:head# 2024-04-25T09:28:14.524 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.557 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d3676469:::benchmark_data_smithi102_474039_object257:head# 2024-04-25T09:28:14.558 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.591 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d54af8ff:::benchmark_data_smithi102_474039_object243:head# 2024-04-25T09:28:14.591 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.624 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d55dc631:::benchmark_data_smithi102_474039_object135:head# 2024-04-25T09:28:14.625 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.658 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d57afc5e:::benchmark_data_smithi102_474039_object239:head# 2024-04-25T09:28:14.658 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.733 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d59a5c6f:::benchmark_data_smithi102_474039_object114:head# 2024-04-25T09:28:14.733 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.794 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d5c6e890:::benchmark_data_smithi102_474039_object55:head# 2024-04-25T09:28:14.794 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.859 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d6176eb6:::benchmark_data_smithi102_474039_object166:head# 2024-04-25T09:28:14.859 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.892 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d7092e79:::benchmark_data_smithi102_474039_object184:head# 2024-04-25T09:28:14.893 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.926 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d82cc540:::benchmark_data_smithi102_474039_object228:head# 2024-04-25T09:28:14.926 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.959 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d8845e1d:::benchmark_data_smithi102_474039_object1:head# 2024-04-25T09:28:14.959 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:14.993 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:d9bfc5ee:::benchmark_data_smithi102_474039_object206:head# 2024-04-25T09:28:14.993 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.026 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:da26776c:::benchmark_data_smithi102_474039_object160:head# 2024-04-25T09:28:15.026 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.060 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:db102a11:::benchmark_data_smithi102_474039_object81:head# 2024-04-25T09:28:15.060 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.102 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:db5c7cb9:::benchmark_data_smithi102_474039_object150:head# 2024-04-25T09:28:15.102 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.144 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dbdbad05:::benchmark_data_smithi102_474039_object101:head# 2024-04-25T09:28:15.144 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.185 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dd4d9c69:::benchmark_data_smithi102_474039_object12:head# 2024-04-25T09:28:15.185 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.227 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dd4e1b82:::benchmark_data_smithi102_474039_object310:head# 2024-04-25T09:28:15.227 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.269 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:df5fb060:::benchmark_data_smithi102_474039_object119:head# 2024-04-25T09:28:15.269 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.310 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dfacd7bf:::benchmark_data_smithi102_474039_object70:head# 2024-04-25T09:28:15.311 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.352 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:dfd9d5b5:::benchmark_data_smithi102_474039_object174:head# 2024-04-25T09:28:15.353 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.394 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e010c870:::benchmark_data_smithi102_474039_object259:head# 2024-04-25T09:28:15.394 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.436 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e186b86e:::benchmark_data_smithi102_474039_object309:head# 2024-04-25T09:28:15.436 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.478 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e56f5a89:::benchmark_data_smithi102_474039_object152:head# 2024-04-25T09:28:15.478 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.519 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e66f5dbd:::benchmark_data_smithi102_474039_object131:head# 2024-04-25T09:28:15.520 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.586 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e6bfbf39:::benchmark_data_smithi102_474039_object237:head# 2024-04-25T09:28:15.587 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.628 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e6d03ffb:::benchmark_data_smithi102_474039_object64:head# 2024-04-25T09:28:15.628 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.670 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e762bddb:::benchmark_data_smithi102_474039_object198:head# 2024-04-25T09:28:15.670 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.711 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e7bfdaaf:::benchmark_data_smithi102_474039_object172:head# 2024-04-25T09:28:15.712 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.754 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e9757145:::benchmark_data_smithi102_474039_object161:head# 2024-04-25T09:28:15.754 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.795 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:e9d173d9:::benchmark_data_smithi102_474039_object308:head# 2024-04-25T09:28:15.796 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.837 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ea1aeea3:::benchmark_data_smithi102_474039_object196:head# 2024-04-25T09:28:15.837 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.879 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:eaacd3fc:::benchmark_data_smithi102_474039_object261:head# 2024-04-25T09:28:15.879 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.921 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:eb6042a7:::benchmark_data_smithi102_474039_object320:head# 2024-04-25T09:28:15.921 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:15.962 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ec593f7d:::benchmark_data_smithi102_474039_object157:head# 2024-04-25T09:28:15.962 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.004 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ed72cdc3:::benchmark_data_smithi102_474039_object266:head# 2024-04-25T09:28:16.004 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.046 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:edfbd108:::benchmark_data_smithi102_474039_object199:head# 2024-04-25T09:28:16.046 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.088 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f033aeef:::benchmark_data_smithi102_474039_object149:head# 2024-04-25T09:28:16.088 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.139 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f08b9ea6:::benchmark_data_smithi102_474039_object87:head# 2024-04-25T09:28:16.139 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.205 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f09ab9bd:::benchmark_data_smithi102_474039_object61:head# 2024-04-25T09:28:16.205 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.266 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f2ba6301:::benchmark_data_smithi102_474039_object30:head# 2024-04-25T09:28:16.266 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.305 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f2f6c7dc:::benchmark_data_smithi102_474039_object192:head# 2024-04-25T09:28:16.305 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.347 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f3685838:::benchmark_data_smithi102_474039_object21:head# 2024-04-25T09:28:16.347 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.389 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f5676a95:::benchmark_data_smithi102_474039_object18:head# 2024-04-25T09:28:16.389 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.439 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f5b6f02e:::benchmark_data_smithi102_474039_object47:head# 2024-04-25T09:28:16.439 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.489 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f71eddbf:::benchmark_data_smithi102_474039_object304:head# 2024-04-25T09:28:16.489 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.531 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f7fc3b27:::benchmark_data_smithi102_474039_object120:head# 2024-04-25T09:28:16.531 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.581 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:f8518d9b:::benchmark_data_smithi102_474039_object44:head# 2024-04-25T09:28:16.581 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.623 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fab39e4b:::benchmark_data_smithi102_474039_object106:head# 2024-04-25T09:28:16.623 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.665 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fb1138dd:::benchmark_data_smithi102_474039_object240:head# 2024-04-25T09:28:16.665 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.732 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fb96aee6:::benchmark_data_smithi102_474039_object245:head# 2024-04-25T09:28:16.732 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.807 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fc30d971:::benchmark_data_smithi102_474039_object58:head# 2024-04-25T09:28:16.807 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:16.865 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fd9e2883:::benchmark_data_smithi102_474039_object38:head# 2024-04-25T09:28:16.866 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:17.007 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fe977974:::benchmark_data_smithi102_474039_object286:head# 2024-04-25T09:28:17.008 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:17.158 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:fede8008:::benchmark_data_smithi102_474039_object62:head# 2024-04-25T09:28:17.158 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:17.266 INFO:tasks.workunit.client.0.smithi102.stdout:Write #1:ff347bd7:::benchmark_data_smithi102_474039_object129:head# 2024-04-25T09:28:17.267 INFO:tasks.workunit.client.0.smithi102.stdout:snapset 0=[]:{} 2024-04-25T09:28:17.300 INFO:tasks.workunit.client.0.smithi102.stdout:write_pg epoch 9 info 1.0( v 9'322 (0'0,9'322] local-lis/les=7/8 n=322 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:28:17.342 INFO:tasks.workunit.client.0.smithi102.stdout:Import successful 2024-04-25T09:28:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:193: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-04-25T09:28:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:28:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:28:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:28:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:28:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:28:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:28:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:28:17.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:28:17.885 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:28:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' 2024-04-25T09:28:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:28:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:28:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:28:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:28:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:28:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:28:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:28:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:28:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:28:17.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:28:17.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:28:17.888 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:28:17.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=afc5def2-9680-456e-9c75-f8f00ce434ef --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.450371/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:28:17.889 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:28:17.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:28:17.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:28:17.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:28:17.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:28:17.921 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:17.935+0000 7f160a2fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:17.924 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:17.939+0000 7f160a2fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:17.925 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:17.939+0000 7f160a2fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:18.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:28:18.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:28:18.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:28:18.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:28:18.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:28:18.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:18.344 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:28:18.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:28:18.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:18.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:18.467 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:18.483+0000 7f160a2fb7c0 -1 Falling back to public interface 2024-04-25T09:28:18.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:19.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:19.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:19.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:28:19.865 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:28:19.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:19.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:20.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:21.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:21.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:21.293 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:28:21.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:28:21.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:21.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:21.590 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:21.606+0000 7f160a2fb7c0 -1 osd.0 18 log_to_monitors true 2024-04-25T09:28:21.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:22.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:22.721 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:22.721 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:28:22.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:28:22.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:22.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:23.162 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [12,18) [v2:127.0.0.1:6802/3860717714,v1:127.0.0.1:6803/3860717714] [v2:127.0.0.1:6804/3860717714,v1:127.0.0.1:6805/3860717714] exists,up b7b813c4-be34-4c65-abf1-0e52eb14287b 2024-04-25T09:28:23.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:28:23.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:28:23.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:28:23.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:195: TEST_import_after_split: wait_for_clean 2024-04-25T09:28:23.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:28:23.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:28:23.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:28:23.163 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:28:23.164 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:28:23.164 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:28:23.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:28:23.164 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:28:23.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:28:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:28:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:28:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:28:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:28:23.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:28:23.267 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:28:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:28:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:28:23.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:28:23.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:28:23.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313219 2024-04-25T09:28:23.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313219 2024-04-25T09:28:23.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-90194313219' 2024-04-25T09:28:23.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:28:23.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:28:23.939 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-90194313219 2024-04-25T09:28:23.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:28:23.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-90194313219 2024-04-25T09:28:23.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:28:23.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313219 2024-04-25T09:28:23.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 90194313219' 2024-04-25T09:28:23.943 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 90194313219 2024-04-25T09:28:23.943 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:28:24.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313219 -lt 90194313219 2024-04-25T09:28:24.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:28:24.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:28:24.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:28:24.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:28:24.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:28:24.959 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:28:24.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:28:24.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:28:24.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:28:24.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:28:24.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:28:25.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:28:25.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:28:25.354 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:28:25.355 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:28:25.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:28:25.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:28:25.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:28:25.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:28:25.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:28:25.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:28:25.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:28:25.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:28:25.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:28:25.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:28:25.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:28:25.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:28:25.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:28:25.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:28:25.981 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:28:25.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:28:25.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:28:25.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:28:25.983 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:28:25.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:28:25.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:28:25.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:28:25.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:28:25.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:28:25.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:28:25.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:28:25.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:28:25.994 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:25.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:28:25.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.450371 2024-04-25T09:28:25.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:28:25.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:28:25.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:28:25.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/pg-split-merge 0 2024-04-25T09:28:25.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:28:25.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:28:25.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:28:25.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:28:25.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:28:25.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:28:25.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:28:25.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:28:25.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:28:25.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:28:25.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:28:25.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:28:26.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:28:26.000 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:28:26.000 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:28:26.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:28:26.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:28:26.001 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:28:26.001 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:28:26.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:28:26.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:28:26.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:28:26.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:28:26.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:26.004 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.450371 2024-04-25T09:28:26.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.450371 2024-04-25T09:28:26.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:28:26.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:28:26.005 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:28:26.006 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:28:26.006 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:28:26.058 INFO:tasks.workunit:Running workunit osd/repeer-on-acting-back.sh... 2024-04-25T09:28:26.058 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/repeer-on-acting-back.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh 2024-04-25T09:28:26.111 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:28:26.116 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:28:26.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:28:26.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:28:26.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:28:26.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:28:26.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:28:26.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:28:26.116 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:28:26.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:28:26.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/repeer-on-acting-back 2024-04-25T09:28:26.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:22: run: local dir=td/repeer-on-acting-back 2024-04-25T09:28:26.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:23: run: shift 2024-04-25T09:28:26.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: export poolname=test 2024-04-25T09:28:26.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: poolname=test 2024-04-25T09:28:26.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: export testobjects=100 2024-04-25T09:28:26.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: testobjects=100 2024-04-25T09:28:26.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: export loglen=12 2024-04-25T09:28:26.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: loglen=12 2024-04-25T09:28:26.117 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: expr 12 / 2 2024-04-25T09:28:26.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: export trim=6 2024-04-25T09:28:26.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: trim=6 2024-04-25T09:28:26.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T09:28:26.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: CEPH_MON=127.0.0.1:7115 2024-04-25T09:28:26.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:30: run: export CEPH_ARGS 2024-04-25T09:28:26.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: uuidgen 2024-04-25T09:28:26.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: CEPH_ARGS+='--fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none ' 2024-04-25T09:28:26.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T09:28:26.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:34: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:28:26.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:36: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:28:26.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: set 2024-04-25T09:28:26.119 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: local funcs=TEST_repeer_on_down_acting_member_coming_back 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:39: run: for func in $funcs 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:40: run: setup td/repeer-on-acting-back 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repeer-on-acting-back 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repeer-on-acting-back 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:28:26.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:28:26.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:28:26.122 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:28:26.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:28:26.123 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:28:26.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:28:26.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:28:26.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:28:26.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:28:26.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:28:26.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:28:26.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:28:26.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:28:26.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:28:26.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:28:26.127 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:28:26.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:26.128 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:28:26.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.477453 2024-04-25T09:28:26.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:28:26.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:28:26.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repeer-on-acting-back 2024-04-25T09:28:26.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:28:26.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:26.129 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:28:26.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.477453 2024-04-25T09:28:26.131 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:28:26.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:28:26.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:28:26.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:28:26.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repeer-on-acting-back 1' TERM HUP INT 2024-04-25T09:28:26.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:41: run: TEST_repeer_on_down_acting_member_coming_back td/repeer-on-acting-back 2024-04-25T09:28:26.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:48: TEST_repeer_on_down_acting_member_coming_back: local dir=td/repeer-on-acting-back 2024-04-25T09:28:26.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:49: TEST_repeer_on_down_acting_member_coming_back: local dummyfile=/etc/fstab 2024-04-25T09:28:26.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:51: TEST_repeer_on_down_acting_member_coming_back: local num_osds=6 2024-04-25T09:28:26.132 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: expr 6 - 1 2024-04-25T09:28:26.132 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: seq 0 5 2024-04-25T09:28:26.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: local 'osds=0 2024-04-25T09:28:26.133 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:28:26.133 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:28:26.133 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T09:28:26.133 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T09:28:26.133 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T09:28:26.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:53: TEST_repeer_on_down_acting_member_coming_back: run_mon td/repeer-on-acting-back a 2024-04-25T09:28:26.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repeer-on-acting-back 2024-04-25T09:28:26.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:28:26.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:28:26.133 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:28:26.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repeer-on-acting-back/a 2024-04-25T09:28:26.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repeer-on-acting-back/a --run-dir=td/repeer-on-acting-back 2024-04-25T09:28:26.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:28:26.649 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:28:26.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:28:26.650 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:28:26.650 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:26.650 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:28:26.650 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:28:26.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repeer-on-acting-back/a '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --mon-cluster-log-file=td/repeer-on-acting-back/log --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:28:26.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:28:26.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:28:26.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:28:26.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:28:26.880 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:28:26.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:28:26.881 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:28:26.881 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:28:26.881 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:28:26.881 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:28:26.881 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:26.881 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:28:26.882 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.477453/ceph-mon.a.asok 2024-04-25T09:28:26.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:28:26.882 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.477453/ceph-mon.a.asok config get fsid 2024-04-25T09:28:26.996 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:28:26.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:28:26.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:28:26.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.477453/ceph-mon.a.asok 2024-04-25T09:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:28:26.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.477453/ceph-mon.a.asok config get mon_host 2024-04-25T09:28:27.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:54: TEST_repeer_on_down_acting_member_coming_back: run_mgr td/repeer-on-acting-back x 2024-04-25T09:28:27.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repeer-on-acting-back 2024-04-25T09:28:27.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:28:27.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:28:27.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:28:27.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repeer-on-acting-back/x 2024-04-25T09:28:27.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:28:28.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:28:28.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:28:28.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:28:28.041 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:28:28.041 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:28.041 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:28:28.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:28:28.041 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:28:28.042 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repeer-on-acting-back/x '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:28:28.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:28:28.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 0 2024-04-25T09:28:28.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:28:28.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:28:28.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:28:28.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:28:28.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/0 2024-04-25T09:28:28.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:28:28.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/0' 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/0/journal' 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:28:28.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/0 2024-04-25T09:28:28.082 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:28:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=992190c0-1c55-499e-8538-4b0f37a40a20 2024-04-25T09:28:28.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 992190c0-1c55-499e-8538-4b0f37a40a20' 2024-04-25T09:28:28.083 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 992190c0-1c55-499e-8538-4b0f37a40a20 2024-04-25T09:28:28.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:28:28.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8IipmtDcsBxAAp0Z3E23f1BG1dSkf3FL+3g== 2024-04-25T09:28:28.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8IipmtDcsBxAAp0Z3E23f1BG1dSkf3FL+3g=="}' 2024-04-25T09:28:28.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 992190c0-1c55-499e-8538-4b0f37a40a20 -i td/repeer-on-acting-back/0/new.json 2024-04-25T09:28:28.414 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:28:28.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/0/new.json 2024-04-25T09:28:28.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8IipmtDcsBxAAp0Z3E23f1BG1dSkf3FL+3g== --osd-uuid 992190c0-1c55-499e-8538-4b0f37a40a20 2024-04-25T09:28:28.469 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:28.482+0000 7f5dd943f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:28.482 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:28.498+0000 7f5dd943f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:28.483 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:28.498+0000 7f5dd943f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:28.484 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:28.498+0000 7f5dd943f7c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:28:28.484 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:28.498+0000 7f5dd943f7c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:28:28.484 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:28.498+0000 7f5dd943f7c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:28:28.522 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:28.538+0000 7f5dd943f7c0 -1 bluestore(td/repeer-on-acting-back/0) _read_fsid unparsable uuid 2024-04-25T09:28:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/0/keyring 2024-04-25T09:28:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:28:35.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:28:35.031 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:28:35.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:28:35.582 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:28:35.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:28:35.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:28:35.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:28:35.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:28:35.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:28:35.622 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:35.634+0000 7f68396ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:35.624 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:35.638+0000 7f68396ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:35.625 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:35.638+0000 7f68396ed7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:36.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:28:36.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:28:36.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:28:36.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:28:36.009 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:28:36.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:28:36.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:36.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:28:36.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:36.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:36.212 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:36.226+0000 7f68396ed7c0 -1 Falling back to public interface 2024-04-25T09:28:36.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:37.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:37.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:37.450 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:28:37.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:28:37.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:37.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:37.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:38.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:38.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:38.866 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:28:38.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:28:38.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:38.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:38.900 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:28:38.914+0000 7f68396ed7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:28:39.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:40.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:40.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:40.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:28:40.429 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:28:40.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:40.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:40.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:41.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:41.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:41.913 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:28:41.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:28:41.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:41.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:42.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:43.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:43.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:43.420 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:28:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:28:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:43.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:43.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:44.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:44.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:44.895 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:28:44.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:28:44.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:44.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:45.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:46.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:46.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:46.379 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:28:46.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:28:46.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:46.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:46.850 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:47.851 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:47.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:47.852 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:28:47.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:28:47.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:47.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:48.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:49.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:49.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:49.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:28:49.281 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:28:49.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:49.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:49.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:50.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:50.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:50.921 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:28:50.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:28:50.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:50.922 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:51.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:52.385 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:28:52.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:52.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:52.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:28:52.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:52.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:53.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:54.258 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:54.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:54.259 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:28:54.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:28:54.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:54.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:54.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:55.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:55.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:55.932 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:28:55.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:28:55.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:55.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:56.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:57.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:57.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:57.615 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:28:57.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:28:57.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:57.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:58.445 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:59.447 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:28:59.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:59.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:59.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:28:59.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:59.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:00.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:01.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:01.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:01.802 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:29:01.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:29:01.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:01.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:02.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:03.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:03.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:03.676 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:29:03.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:29:03.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:03.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:04.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:05.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:05.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:05.283 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:29:05.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:29:05.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:05.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:29:05.730 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3214025673,v1:127.0.0.1:6803/3214025673] [v2:127.0.0.1:6804/3214025673,v1:127.0.0.1:6805/3214025673] exists,up 992190c0-1c55-499e-8538-4b0f37a40a20 2024-04-25T09:29:05.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:29:05.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:29:05.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:29:05.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:29:05.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 1 2024-04-25T09:29:05.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:29:05.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:29:05.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:29:05.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:29:05.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/1 2024-04-25T09:29:05.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:29:05.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:29:05.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:29:05.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:29:05.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/1' 2024-04-25T09:29:05.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/1/journal' 2024-04-25T09:29:05.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:29:05.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:29:05.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:29:05.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:29:05.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:29:05.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:29:05.737 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:29:05.737 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:05.738 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:29:05.738 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:29:05.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:29:05.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:29:05.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:29:05.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:29:05.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:29:05.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:29:05.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:29:05.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:29:05.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:29:05.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:29:05.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:29:05.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:29:05.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/1 2024-04-25T09:29:05.740 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:29:05.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b58f0f2e-23c4-4daf-b70a-1fff0f6da846 2024-04-25T09:29:05.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 b58f0f2e-23c4-4daf-b70a-1fff0f6da846' 2024-04-25T09:29:05.741 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 b58f0f2e-23c4-4daf-b70a-1fff0f6da846 2024-04-25T09:29:05.741 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:29:05.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhIipml6QQLhAAGh5tmyyB6PmM7OGWT+AKzA== 2024-04-25T09:29:05.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhIipml6QQLhAAGh5tmyyB6PmM7OGWT+AKzA=="}' 2024-04-25T09:29:05.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b58f0f2e-23c4-4daf-b70a-1fff0f6da846 -i td/repeer-on-acting-back/1/new.json 2024-04-25T09:29:06.244 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:29:06.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/1/new.json 2024-04-25T09:29:06.261 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhIipml6QQLhAAGh5tmyyB6PmM7OGWT+AKzA== --osd-uuid b58f0f2e-23c4-4daf-b70a-1fff0f6da846 2024-04-25T09:29:06.307 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:06.322+0000 7f70582727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:06.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:06.330+0000 7f70582727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:06.318 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:06.334+0000 7f70582727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:06.318 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:06.334+0000 7f70582727c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:29:06.318 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:06.334+0000 7f70582727c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:29:06.318 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:06.334+0000 7f70582727c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:29:06.404 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:06.422+0000 7f70582727c0 -1 bluestore(td/repeer-on-acting-back/1) _read_fsid unparsable uuid 2024-04-25T09:29:13.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/1/keyring 2024-04-25T09:29:13.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:29:13.020 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:29:13.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:29:13.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:29:13.600 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:29:13.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:29:13.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:29:13.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:29:13.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:29:13.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:29:13.639 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:13.653+0000 7ff16fbc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:13.647 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:13.661+0000 7ff16fbc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:13.648 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:13.661+0000 7ff16fbc77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:14.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:29:14.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:29:14.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:29:14.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:29:14.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:29:14.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:14.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:29:14.019 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:29:14.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:14.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:14.220 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:14.233+0000 7ff16fbc77c0 -1 Falling back to public interface 2024-04-25T09:29:14.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:15.466 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:15.466 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:29:15.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:15.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:29:15.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:15.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:15.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:16.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:16.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:16.902 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:29:16.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:29:16.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:16.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:16.938 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:16.953+0000 7ff16fbc77c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:29:17.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:18.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:18.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:18.512 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:29:18.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:29:18.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:18.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:19.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:20.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:20.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:20.150 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:29:20.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:29:20.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:20.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:20.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:21.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:21.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:21.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:29:21.566 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:29:21.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:21.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:22.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:23.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:23.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:23.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:29:23.102 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:29:23.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:23.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:23.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:24.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:24.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:24.600 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:29:24.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:29:24.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:24.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:25.244 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:26.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:26.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:26.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:29:26.247 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:29:26.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:26.247 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:26.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:27.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:27.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:27.727 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:29:27.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:29:27.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:27.728 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:28.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:29.336 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:29.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:29.337 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:29:29.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:29:29.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:29.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:29.869 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:30.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:30.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:30.871 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:29:30.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:29:30.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:30.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:31.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:32.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:32.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:32.407 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:29:32.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:29:32.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:32.408 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:33.362 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:34.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:34.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:34.363 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:29:34.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:29:34.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:34.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:34.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:35.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:35.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:35.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:29:35.904 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:29:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:35.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:36.692 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:37.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:37.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:37.694 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:29:37.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:29:37.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:37.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:38.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:39.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:39.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:39.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:29:39.533 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:29:39.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:39.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:40.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:41.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:41.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:41.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:29:41.473 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:29:41.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:41.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:42.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:43.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:29:43.637 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:29:43.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:43.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:44.064 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1687422604,v1:127.0.0.1:6811/1687422604] [v2:127.0.0.1:6812/1687422604,v1:127.0.0.1:6813/1687422604] exists,up b58f0f2e-23c4-4daf-b70a-1fff0f6da846 2024-04-25T09:29:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:29:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:29:44.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 2 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/2 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-04-25T09:29:44.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:29:44.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:29:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:29:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:29:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:29:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:29:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:29:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:29:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:29:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:29:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:29:44.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/2 2024-04-25T09:29:44.071 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:29:44.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a5080eaa-8b5e-48b8-b649-ac0c77890f4a 2024-04-25T09:29:44.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a5080eaa-8b5e-48b8-b649-ac0c77890f4a' 2024-04-25T09:29:44.072 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 a5080eaa-8b5e-48b8-b649-ac0c77890f4a 2024-04-25T09:29:44.072 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:29:44.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIIipmR43YBhAAC1VbZGXDHREaV3Y1chTfDA== 2024-04-25T09:29:44.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIIipmR43YBhAAC1VbZGXDHREaV3Y1chTfDA=="}' 2024-04-25T09:29:44.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a5080eaa-8b5e-48b8-b649-ac0c77890f4a -i td/repeer-on-acting-back/2/new.json 2024-04-25T09:29:44.570 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:29:44.586 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/2/new.json 2024-04-25T09:29:44.587 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIIipmR43YBhAAC1VbZGXDHREaV3Y1chTfDA== --osd-uuid a5080eaa-8b5e-48b8-b649-ac0c77890f4a 2024-04-25T09:29:44.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:44.633+0000 7f9e5de1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:44.657 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:44.669+0000 7f9e5de1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:44.658 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:44.673+0000 7f9e5de1b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:44.659 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:44.673+0000 7f9e5de1b7c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:29:44.659 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:44.673+0000 7f9e5de1b7c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:29:44.659 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:44.673+0000 7f9e5de1b7c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:29:44.687 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:44.701+0000 7f9e5de1b7c0 -1 bluestore(td/repeer-on-acting-back/2) _read_fsid unparsable uuid 2024-04-25T09:29:51.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/2/keyring 2024-04-25T09:29:51.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:29:51.373 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T09:29:51.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:29:51.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:29:51.928 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:29:51.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:29:51.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:29:51.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:29:51.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:29:51.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:29:51.960 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:51.973+0000 7fe0f521a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:51.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:52.009+0000 7fe0f521a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:51.996 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:52.009+0000 7fe0f521a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:52.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:29:52.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:29:52.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:29:52.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:29:52.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:29:52.378 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:52.378 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:29:52.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:29:52.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:52.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:52.571 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:52.585+0000 7fe0f521a7c0 -1 Falling back to public interface 2024-04-25T09:29:52.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:53.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:53.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:53.813 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:29:53.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:29:53.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:53.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:54.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:55.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:55.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:55.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:29:55.235 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:29:55.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:55.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:55.266 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:29:55.281+0000 7fe0f521a7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:29:55.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:56.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:56.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:56.755 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:29:56.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:29:56.755 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:56.756 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:57.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:58.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:58.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:58.340 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:29:58.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:29:58.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:58.340 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:58.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:59.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:59.773 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:59.774 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:29:59.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:29:59.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:59.775 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:00.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:01.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:01.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:01.355 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:30:01.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:30:01.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:01.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:01.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:02.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:02.786 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:02.786 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:30:02.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:30:02.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:02.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:03.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:04.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:04.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:04.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:30:04.339 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:30:04.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:04.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:04.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:05.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:05.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:05.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:30:05.831 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:30:05.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:05.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:06.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:07.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:07.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:07.359 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:30:07.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:30:07.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:07.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:07.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:08.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:08.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:08.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:30:08.881 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:30:08.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:08.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:09.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:10.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:10.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:10.457 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:30:10.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:30:10.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:10.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:11.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:12.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:12.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:12.495 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:30:12.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:30:12.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:12.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:13.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:14.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:14.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:14.441 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:30:14.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:30:14.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:14.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:15.262 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:16.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:16.264 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:16.264 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:30:16.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:30:16.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:16.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:17.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:18.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:18.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:18.293 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:30:18.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:30:18.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:18.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:19.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:20.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:20.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:20.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:30:20.665 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:30:20.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:20.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:21.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:22.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:22.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:22.348 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:30:22.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:30:22.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:22.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:22.942 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1163549235,v1:127.0.0.1:6819/1163549235] [v2:127.0.0.1:6820/1163549235,v1:127.0.0.1:6821/1163549235] exists,up a5080eaa-8b5e-48b8-b649-ac0c77890f4a 2024-04-25T09:30:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:30:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:30:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:30:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:30:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 3 2024-04-25T09:30:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:30:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:30:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T09:30:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:30:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/3 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/3' 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/3/journal' 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:30:22.947 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:30:22.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:30:22.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/3 2024-04-25T09:30:22.949 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:30:22.949 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3d828bbd-b97d-40e5-a8df-9efb2ad99d2f 2024-04-25T09:30:22.950 INFO:tasks.workunit.client.0.smithi102.stdout:add osd3 3d828bbd-b97d-40e5-a8df-9efb2ad99d2f 2024-04-25T09:30:22.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 3d828bbd-b97d-40e5-a8df-9efb2ad99d2f' 2024-04-25T09:30:22.950 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:30:22.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCuIipmzykaOxAAC0Wi1nAWNsJuoCGmnaZZfQ== 2024-04-25T09:30:22.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCuIipmzykaOxAAC0Wi1nAWNsJuoCGmnaZZfQ=="}' 2024-04-25T09:30:22.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3d828bbd-b97d-40e5-a8df-9efb2ad99d2f -i td/repeer-on-acting-back/3/new.json 2024-04-25T09:30:23.494 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:30:23.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/3/new.json 2024-04-25T09:30:23.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCuIipmzykaOxAAC0Wi1nAWNsJuoCGmnaZZfQ== --osd-uuid 3d828bbd-b97d-40e5-a8df-9efb2ad99d2f 2024-04-25T09:30:23.549 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:23.560+0000 7f8e52bd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:23.577 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:23.588+0000 7f8e52bd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:23.578 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:23.592+0000 7f8e52bd07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:23.592+0000 7f8e52bd07c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:30:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:23.592+0000 7f8e52bd07c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:30:23.579 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:23.592+0000 7f8e52bd07c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:30:23.617 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:23.628+0000 7f8e52bd07c0 -1 bluestore(td/repeer-on-acting-back/3) _read_fsid unparsable uuid 2024-04-25T09:30:30.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/3/keyring 2024-04-25T09:30:30.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:30:30.387 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd3 key to auth repository 2024-04-25T09:30:30.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T09:30:30.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:30:30.977 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.3 2024-04-25T09:30:30.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T09:30:30.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:30:30.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:30:30.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:30:30.978 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:30:31.015 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:31.028+0000 7f32fe41d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:31.062 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:31.072+0000 7f32fe41d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:31.062 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:31.072+0000 7f32fe41d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:31.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T09:30:31.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:30:31.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T09:30:31.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:30:31.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:30:31.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:31.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:30:31.374 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:30:31.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:31.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:31.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:31.632+0000 7f32fe41d7c0 -1 Falling back to public interface 2024-04-25T09:30:31.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:32.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:32.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:32.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:30:32.809 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:30:32.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:32.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:33.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:34.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:34.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:34.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:30:34.223 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:30:34.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:34.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:34.234 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:30:34.248+0000 7f32fe41d7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T09:30:34.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:35.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:35.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:35.831 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:30:35.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:30:35.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:35.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:36.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:37.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:37.434 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:30:37.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:37.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:30:37.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:37.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:38.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:39.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:39.018 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:30:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:30:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:39.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:39.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:40.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:40.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:40.548 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:30:40.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:30:40.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:40.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:40.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:41.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:41.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:41.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:30:41.994 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:30:41.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:41.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:42.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:43.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:43.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:43.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:30:43.519 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:30:43.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:43.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:44.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:45.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:45.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:45.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:30:45.007 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:30:45.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:45.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:45.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:46.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:46.525 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:46.525 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:30:46.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:30:46.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:46.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:46.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:47.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:47.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:47.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:30:47.968 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:30:47.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:47.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:48.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:49.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:49.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:49.504 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:30:49.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:30:49.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:49.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:50.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:51.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:51.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:51.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:30:51.065 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:30:51.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:51.066 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:51.931 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:52.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:52.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:52.933 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:30:52.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:30:52.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:52.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:53.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:54.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:54.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:54.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:30:54.844 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:30:54.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:54.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:55.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:56.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:56.365 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:56.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:30:56.366 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:30:56.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:56.366 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:57.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:58.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:58.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:58.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:30:58.190 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:30:58.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:58.191 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:30:59.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:00.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:00.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:00.014 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:31:00.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:31:00.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:00.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:31:00.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:01.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:01.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:01.539 INFO:tasks.workunit.client.0.smithi102.stdout:19 2024-04-25T09:31:01.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:31:01.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:01.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:31:02.031 INFO:tasks.workunit.client.0.smithi102.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2081957381,v1:127.0.0.1:6827/2081957381] [v2:127.0.0.1:6828/2081957381,v1:127.0.0.1:6829/2081957381] exists,up 3d828bbd-b97d-40e5-a8df-9efb2ad99d2f 2024-04-25T09:31:02.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 4 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/4 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:31:02.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:31:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/4' 2024-04-25T09:31:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/4/journal' 2024-04-25T09:31:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:31:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:31:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:31:02.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:31:02.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:31:02.035 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:31:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:31:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:31:02.035 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:31:02.036 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:31:02.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:31:02.036 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/4 2024-04-25T09:31:02.037 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:31:02.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7eb85347-a712-4474-931c-0328ada317ca 2024-04-25T09:31:02.038 INFO:tasks.workunit.client.0.smithi102.stdout:add osd4 7eb85347-a712-4474-931c-0328ada317ca 2024-04-25T09:31:02.038 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 7eb85347-a712-4474-931c-0328ada317ca' 2024-04-25T09:31:02.038 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:31:02.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWIipmFkDmBBAAIB1BvGsihRB/5bPbLXgQHg== 2024-04-25T09:31:02.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWIipmFkDmBBAAIB1BvGsihRB/5bPbLXgQHg=="}' 2024-04-25T09:31:02.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7eb85347-a712-4474-931c-0328ada317ca -i td/repeer-on-acting-back/4/new.json 2024-04-25T09:31:02.563 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:31:02.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/4/new.json 2024-04-25T09:31:02.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWIipmFkDmBBAAIB1BvGsihRB/5bPbLXgQHg== --osd-uuid 7eb85347-a712-4474-931c-0328ada317ca 2024-04-25T09:31:02.620 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:02.635+0000 7f0c75a087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:02.659 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:02.675+0000 7f0c75a087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:02.660 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:02.675+0000 7f0c75a087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:02.660 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:02.675+0000 7f0c75a087c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:31:02.661 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:02.675+0000 7f0c75a087c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:31:02.661 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:02.675+0000 7f0c75a087c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:31:02.705 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:02.719+0000 7f0c75a087c0 -1 bluestore(td/repeer-on-acting-back/4) _read_fsid unparsable uuid 2024-04-25T09:31:08.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/4/keyring 2024-04-25T09:31:08.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:31:08.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T09:31:08.628 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd4 key to auth repository 2024-04-25T09:31:08.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:31:09.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T09:31:09.189 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.4 2024-04-25T09:31:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:31:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:31:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:31:09.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:31:09.227 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:09.243+0000 7fec3970a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:09.280 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:09.295+0000 7fec3970a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:09.281 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:09.299+0000 7fec3970a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:09.583 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T09:31:09.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:31:09.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T09:31:09.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:31:09.584 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:31:09.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:31:09.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:09.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:31:09.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:09.585 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:09.840 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:09.855+0000 7fec3970a7c0 -1 Falling back to public interface 2024-04-25T09:31:10.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:11.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:11.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:11.013 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:31:11.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:31:11.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:11.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:11.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:12.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:12.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:12.439 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:31:12.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:31:12.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:12.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:12.510 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:12.527+0000 7fec3970a7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T09:31:12.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:13.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:13.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:13.890 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:31:13.891 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:31:13.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:13.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:14.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:15.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:15.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:15.510 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:31:15.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:31:15.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:15.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:15.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:16.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:16.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:16.940 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:31:16.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:31:16.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:16.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:17.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:18.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:18.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:18.449 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:31:18.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:31:18.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:18.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:18.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:19.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:19.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:19.956 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:31:19.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:31:19.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:19.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:20.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:21.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:21.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:21.512 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:31:21.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:31:21.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:21.513 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:21.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:22.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:22.946 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:31:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:31:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:23.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:24.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:24.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:24.462 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:31:24.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:31:24.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:24.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:24.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:25.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:25.888 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:31:25.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:25.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:31:25.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:25.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:26.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:27.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:27.591 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:27.591 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:31:27.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:31:27.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:27.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:28.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:29.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:29.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:29.456 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:31:29.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:31:29.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:29.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:30.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:31.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:31.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:31.439 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:31:31.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:31:31.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:31.439 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:31.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:32.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:32.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:32.979 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:31:32.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:31:32.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:32.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:33.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:34.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:34.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:34.672 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:31:34.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:31:34.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:34.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:35.776 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:36.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:36.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:36.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:31:36.778 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:31:36.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:36.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:31:37.317 INFO:tasks.workunit.client.0.smithi102.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2936709117,v1:127.0.0.1:6835/2936709117] [v2:127.0.0.1:6836/2936709117,v1:127.0.0.1:6837/2936709117] exists,up 7eb85347-a712-4474-931c-0328ada317ca 2024-04-25T09:31:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:31:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:31:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:31:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:31:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 5 2024-04-25T09:31:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:31:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:31:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T09:31:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:31:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/5 2024-04-25T09:31:37.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/5' 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/5/journal' 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:31:37.321 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:31:37.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:31:37.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/5 2024-04-25T09:31:37.323 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:31:37.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=efafcb44-96fd-49c4-bb36-5a0b620db7ac 2024-04-25T09:31:37.323 INFO:tasks.workunit.client.0.smithi102.stdout:add osd5 efafcb44-96fd-49c4-bb36-5a0b620db7ac 2024-04-25T09:31:37.324 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 efafcb44-96fd-49c4-bb36-5a0b620db7ac' 2024-04-25T09:31:37.324 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:31:37.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5IipmOveQFRAAPVdOXqlENDmR1ZVIaG1EVw== 2024-04-25T09:31:37.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5IipmOveQFRAAPVdOXqlENDmR1ZVIaG1EVw=="}' 2024-04-25T09:31:37.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new efafcb44-96fd-49c4-bb36-5a0b620db7ac -i td/repeer-on-acting-back/5/new.json 2024-04-25T09:31:37.903 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:31:37.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/5/new.json 2024-04-25T09:31:37.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5IipmOveQFRAAPVdOXqlENDmR1ZVIaG1EVw== --osd-uuid efafcb44-96fd-49c4-bb36-5a0b620db7ac 2024-04-25T09:31:37.960 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:37.974+0000 7f723ac377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:37.980 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:37.994+0000 7f723ac377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:37.982 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:37.998+0000 7f723ac377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:37.982 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:37.998+0000 7f723ac377c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:31:37.982 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:37.998+0000 7f723ac377c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:31:37.982 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:37.998+0000 7f723ac377c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:31:38.022 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:38.034+0000 7f723ac377c0 -1 bluestore(td/repeer-on-acting-back/5) _read_fsid unparsable uuid 2024-04-25T09:31:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/5/keyring 2024-04-25T09:31:44.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:31:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T09:31:44.451 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd5 key to auth repository 2024-04-25T09:31:44.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:31:45.026 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.5 2024-04-25T09:31:45.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T09:31:45.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:31:45.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:31:45.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:31:45.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:31:45.064 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:45.078+0000 7f067ed867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:45.066 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:45.082+0000 7f067ed867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:45.067 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:45.082+0000 7f067ed867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:45.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T09:31:45.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:31:45.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T09:31:45.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:31:45.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:31:45.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:45.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:31:45.444 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:31:45.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:45.444 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:31:45.627 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:45.642+0000 7f067ed867c0 -1 Falling back to public interface 2024-04-25T09:31:45.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:46.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:46.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:46.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:31:46.912 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:31:46.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:46.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:31:47.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:48.211 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:31:48.226+0000 7f067ed867c0 -1 osd.5 0 log_to_monitors true 2024-04-25T09:31:48.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:48.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:48.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:31:48.344 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:31:48.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:48.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:31:49.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:50.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:50.006 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:31:50.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:50.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:31:50.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:50.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:31:50.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:51.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:51.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:51.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:31:51.448 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:31:51.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:51.449 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:31:51.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:52.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:52.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:52.900 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:31:52.900 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:31:52.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:52.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:31:53.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:54.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:54.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:54.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:31:54.455 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:31:54.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:54.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:31:54.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:55.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:55.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:55.897 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:31:55.897 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:31:55.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:55.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:31:56.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:57.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:57.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:57.388 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:31:57.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:31:57.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:57.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:31:58.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:59.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:59.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:59.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:31:59.061 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:31:59.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:59.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:31:59.577 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:00.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:32:00.581 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:32:00.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:00.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:32:01.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:02.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:02.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:02.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:32:02.283 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:32:02.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:02.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:32:02.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:03.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:03.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:03.859 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:32:03.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:32:03.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:03.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:32:04.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:05.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:05.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:05.783 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:32:05.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:32:05.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:05.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:32:06.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:07.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:07.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:07.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:32:07.560 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:32:07.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:07.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:32:08.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:09.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:09.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:09.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:32:09.359 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:32:09.359 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:09.360 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:32:10.139 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:11.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:11.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:11.141 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:32:11.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:32:11.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:11.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:32:11.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:12.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:12.655 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:32:12.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:32:12.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:12.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:32:13.077 INFO:tasks.workunit.client.0.smithi102.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/626646153,v1:127.0.0.1:6843/626646153] [v2:127.0.0.1:6844/626646153,v1:127.0.0.1:6845/626646153] exists,up efafcb44-96fd-49c4-bb36-5a0b620db7ac 2024-04-25T09:32:13.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:32:13.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:32:13.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:32:13.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:60: TEST_repeer_on_down_acting_member_coming_back: create_pool test 1 1 2024-04-25T09:32:13.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T09:32:13.631 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T09:32:13.646 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:32:14.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:61: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test size 3 2024-04-25T09:32:15.655 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 size to 3 2024-04-25T09:32:15.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:62: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test min_size 2 2024-04-25T09:32:16.879 INFO:tasks.workunit.client.0.smithi102.stderr:set pool 1 min_size to 2 2024-04-25T09:32:16.902 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: ceph pg dump pools -f json 2024-04-25T09:32:16.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq .pool_stats 2024-04-25T09:32:16.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq '.[].poolid' 2024-04-25T09:32:17.366 INFO:tasks.workunit.client.0.smithi102.stderr:dumped pools 2024-04-25T09:32:17.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: local poolid=1 2024-04-25T09:32:17.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:64: TEST_repeer_on_down_acting_member_coming_back: local pgid=1.0 2024-04-25T09:32:17.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:67: TEST_repeer_on_down_acting_member_coming_back: ceph osd set-require-min-compat-client luminous 2024-04-25T09:32:18.084 INFO:tasks.workunit.client.0.smithi102.stderr:set require_min_compat_client to luminous 2024-04-25T09:32:18.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:69: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 2 3 2024-04-25T09:32:19.139 INFO:tasks.workunit.client.0.smithi102.stderr:set 1.0 pg_upmap mapping to [1,2,3] 2024-04-25T09:32:19.164 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:71: TEST_repeer_on_down_acting_member_coming_back: flush_pg_stats 2024-04-25T09:32:19.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:32:19.165 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:32:19.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:32:19.662 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:32:19.662 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:32:19.662 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T09:32:19.662 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T09:32:19.662 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T09:32:19.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:32:19.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:19.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:32:19.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836528 2024-04-25T09:32:19.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836528 2024-04-25T09:32:19.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836528' 2024-04-25T09:32:19.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:19.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:32:19.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672999 2024-04-25T09:32:19.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672999 2024-04-25T09:32:19.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836528 1-42949672999' 2024-04-25T09:32:19.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:19.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:32:20.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509475 2024-04-25T09:32:20.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509475 2024-04-25T09:32:20.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836528 1-42949672999 2-64424509475' 2024-04-25T09:32:20.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:20.158 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:32:20.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345948 2024-04-25T09:32:20.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345948 2024-04-25T09:32:20.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836528 1-42949672999 2-64424509475 3-85899345948' 2024-04-25T09:32:20.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:20.319 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:32:20.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182414 2024-04-25T09:32:20.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182414 2024-04-25T09:32:20.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836528 1-42949672999 2-64424509475 3-85899345948 4-107374182414' 2024-04-25T09:32:20.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:20.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:32:20.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018888 2024-04-25T09:32:20.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018888 2024-04-25T09:32:20.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836528 1-42949672999 2-64424509475 3-85899345948 4-107374182414 5-128849018888' 2024-04-25T09:32:20.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:20.629 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836528 2024-04-25T09:32:20.630 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:20.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:32:20.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836528 2024-04-25T09:32:20.631 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:20.632 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836528 2024-04-25T09:32:20.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836528 2024-04-25T09:32:20.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836528' 2024-04-25T09:32:20.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:32:21.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836529 -lt 21474836528 2024-04-25T09:32:21.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:21.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672999 2024-04-25T09:32:21.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:21.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:32:21.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672999 2024-04-25T09:32:21.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:21.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672999 2024-04-25T09:32:21.053 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949672999 2024-04-25T09:32:21.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2024-04-25T09:32:21.053 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:32:21.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672999 -lt 42949672999 2024-04-25T09:32:21.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:21.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509475 2024-04-25T09:32:21.476 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:21.477 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:32:21.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509475 2024-04-25T09:32:21.478 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:21.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509475 2024-04-25T09:32:21.479 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509475 2024-04-25T09:32:21.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-04-25T09:32:21.479 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:32:21.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509475 -lt 64424509475 2024-04-25T09:32:21.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:21.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345948 2024-04-25T09:32:21.927 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:21.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:32:21.928 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345948 2024-04-25T09:32:21.929 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:21.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345948 2024-04-25T09:32:21.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345948' 2024-04-25T09:32:21.930 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345948 2024-04-25T09:32:21.931 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:32:22.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345948 -lt 85899345948 2024-04-25T09:32:22.377 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:22.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182414 2024-04-25T09:32:22.379 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:22.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:32:22.381 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182414 2024-04-25T09:32:22.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:22.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182414 2024-04-25T09:32:22.383 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182414 2024-04-25T09:32:22.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2024-04-25T09:32:22.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:32:22.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182414 -lt 107374182414 2024-04-25T09:32:22.789 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:22.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:22.790 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018888 2024-04-25T09:32:22.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:32:22.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018888 2024-04-25T09:32:22.793 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:22.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018888 2024-04-25T09:32:22.794 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018888 2024-04-25T09:32:22.795 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2024-04-25T09:32:22.795 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:32:23.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018888 -lt 128849018888 2024-04-25T09:32:23.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:72: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:32:23.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:32:23.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:32:23.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:32:23.215 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:32:23.215 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:32:23.215 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:32:23.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:32:23.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:32:23.216 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:32:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:32:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:32:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:32:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:32:23.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:32:23.318 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:32:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:32:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:32:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:32:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T09:32:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T09:32:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T09:32:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:32:23.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:23.721 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:32:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836531 2024-04-25T09:32:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836531 2024-04-25T09:32:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531' 2024-04-25T09:32:23.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:23.908 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:32:24.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673002 2024-04-25T09:32:24.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673002 2024-04-25T09:32:24.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531 1-42949673002' 2024-04-25T09:32:24.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:24.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:32:24.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509478 2024-04-25T09:32:24.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509478 2024-04-25T09:32:24.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531 1-42949673002 2-64424509478' 2024-04-25T09:32:24.203 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:24.204 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:32:24.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345951 2024-04-25T09:32:24.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345951 2024-04-25T09:32:24.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531 1-42949673002 2-64424509478 3-85899345951' 2024-04-25T09:32:24.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:24.352 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:32:24.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182417 2024-04-25T09:32:24.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182417 2024-04-25T09:32:24.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531 1-42949673002 2-64424509478 3-85899345951 4-107374182417' 2024-04-25T09:32:24.508 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:24.508 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:32:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018891 2024-04-25T09:32:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018891 2024-04-25T09:32:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836531 1-42949673002 2-64424509478 3-85899345951 4-107374182417 5-128849018891' 2024-04-25T09:32:24.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:24.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836531 2024-04-25T09:32:24.665 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:24.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:32:24.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836531 2024-04-25T09:32:24.666 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:24.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836531 2024-04-25T09:32:24.668 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836531 2024-04-25T09:32:24.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-04-25T09:32:24.668 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:32:25.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836531 -lt 21474836531 2024-04-25T09:32:25.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:25.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:25.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673002 2024-04-25T09:32:25.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:32:25.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673002 2024-04-25T09:32:25.079 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:25.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673002 2024-04-25T09:32:25.080 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673002 2024-04-25T09:32:25.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2024-04-25T09:32:25.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:32:25.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673002 -lt 42949673002 2024-04-25T09:32:25.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:25.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509478 2024-04-25T09:32:25.524 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:25.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:32:25.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509478 2024-04-25T09:32:25.527 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:25.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509478 2024-04-25T09:32:25.528 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509478 2024-04-25T09:32:25.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-04-25T09:32:25.529 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:32:25.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509478 -lt 64424509478 2024-04-25T09:32:25.979 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:25.980 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345951 2024-04-25T09:32:25.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:25.983 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:32:25.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345951 2024-04-25T09:32:25.984 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:25.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345951 2024-04-25T09:32:25.985 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345951 2024-04-25T09:32:25.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2024-04-25T09:32:25.985 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:32:26.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345951 -lt 85899345951 2024-04-25T09:32:26.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:26.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182417 2024-04-25T09:32:26.458 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:26.460 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:32:26.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182417 2024-04-25T09:32:26.461 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:26.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182417 2024-04-25T09:32:26.462 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182417 2024-04-25T09:32:26.463 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2024-04-25T09:32:26.463 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:32:26.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182417 -lt 107374182417 2024-04-25T09:32:26.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:26.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:26.879 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018891 2024-04-25T09:32:26.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:32:26.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018891 2024-04-25T09:32:26.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:26.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018891 2024-04-25T09:32:26.882 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018891 2024-04-25T09:32:26.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018891' 2024-04-25T09:32:26.883 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:32:27.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018891 -lt 128849018891 2024-04-25T09:32:27.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:32:27.312 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:32:27.313 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:32:27.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:32:27.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:32:27.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:32:27.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:32:27.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:32:27.860 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:32:27.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:32:27.861 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:32:28.310 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:32:28.310 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:32:28.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:32:28.311 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:32:28.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:32:28.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:32:28.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:32:28.846 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:74: TEST_repeer_on_down_acting_member_coming_back: echo 'writing initial objects' 2024-04-25T09:32:28.847 INFO:tasks.workunit.client.0.smithi102.stdout:writing initial objects 2024-04-25T09:32:28.847 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: seq 1 100 2024-04-25T09:32:28.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:28.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_1 /etc/fstab 2024-04-25T09:32:29.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:29.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_2 /etc/fstab 2024-04-25T09:32:29.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:29.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_3 /etc/fstab 2024-04-25T09:32:29.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:29.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_4 /etc/fstab 2024-04-25T09:32:29.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:29.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_5 /etc/fstab 2024-04-25T09:32:29.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:29.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_6 /etc/fstab 2024-04-25T09:32:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:29.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_7 /etc/fstab 2024-04-25T09:32:30.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:30.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_8 /etc/fstab 2024-04-25T09:32:30.279 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:30.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_9 /etc/fstab 2024-04-25T09:32:30.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:30.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_10 /etc/fstab 2024-04-25T09:32:30.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:30.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_11 /etc/fstab 2024-04-25T09:32:30.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:30.806 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_12 /etc/fstab 2024-04-25T09:32:30.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:30.975 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_13 /etc/fstab 2024-04-25T09:32:31.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:31.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_14 /etc/fstab 2024-04-25T09:32:31.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:31.335 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_15 /etc/fstab 2024-04-25T09:32:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:31.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_16 /etc/fstab 2024-04-25T09:32:31.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:31.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_17 /etc/fstab 2024-04-25T09:32:31.853 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:31.854 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_18 /etc/fstab 2024-04-25T09:32:32.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:32.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_19 /etc/fstab 2024-04-25T09:32:32.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:32.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_20 /etc/fstab 2024-04-25T09:32:32.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:32.374 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_21 /etc/fstab 2024-04-25T09:32:32.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:32.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_22 /etc/fstab 2024-04-25T09:32:32.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:32.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_23 /etc/fstab 2024-04-25T09:32:32.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:32.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_24 /etc/fstab 2024-04-25T09:32:33.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:33.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_25 /etc/fstab 2024-04-25T09:32:33.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:33.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_26 /etc/fstab 2024-04-25T09:32:33.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:33.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_27 /etc/fstab 2024-04-25T09:32:33.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:33.659 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_28 /etc/fstab 2024-04-25T09:32:33.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:33.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_29 /etc/fstab 2024-04-25T09:32:33.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:33.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_30 /etc/fstab 2024-04-25T09:32:34.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:34.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_31 /etc/fstab 2024-04-25T09:32:34.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:34.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_32 /etc/fstab 2024-04-25T09:32:34.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:34.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_33 /etc/fstab 2024-04-25T09:32:34.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:34.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_34 /etc/fstab 2024-04-25T09:32:35.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:35.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_35 /etc/fstab 2024-04-25T09:32:35.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:35.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_36 /etc/fstab 2024-04-25T09:32:35.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:35.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_37 /etc/fstab 2024-04-25T09:32:35.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:35.599 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_38 /etc/fstab 2024-04-25T09:32:35.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:35.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_39 /etc/fstab 2024-04-25T09:32:35.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:35.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_40 /etc/fstab 2024-04-25T09:32:36.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:36.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_41 /etc/fstab 2024-04-25T09:32:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:36.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_42 /etc/fstab 2024-04-25T09:32:36.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:36.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_43 /etc/fstab 2024-04-25T09:32:36.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:36.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_44 /etc/fstab 2024-04-25T09:32:36.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:36.805 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_45 /etc/fstab 2024-04-25T09:32:36.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:36.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_46 /etc/fstab 2024-04-25T09:32:37.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:37.132 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_47 /etc/fstab 2024-04-25T09:32:37.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:37.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_48 /etc/fstab 2024-04-25T09:32:37.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:37.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_49 /etc/fstab 2024-04-25T09:32:37.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:37.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_50 /etc/fstab 2024-04-25T09:32:37.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:37.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_51 /etc/fstab 2024-04-25T09:32:38.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:38.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_52 /etc/fstab 2024-04-25T09:32:38.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:38.188 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_53 /etc/fstab 2024-04-25T09:32:38.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:38.380 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_54 /etc/fstab 2024-04-25T09:32:38.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:38.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_55 /etc/fstab 2024-04-25T09:32:38.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:38.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_56 /etc/fstab 2024-04-25T09:32:38.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:38.883 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_57 /etc/fstab 2024-04-25T09:32:39.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:39.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_58 /etc/fstab 2024-04-25T09:32:39.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:39.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_59 /etc/fstab 2024-04-25T09:32:39.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:39.438 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_60 /etc/fstab 2024-04-25T09:32:39.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:39.604 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_61 /etc/fstab 2024-04-25T09:32:39.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:39.779 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_62 /etc/fstab 2024-04-25T09:32:39.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:39.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_63 /etc/fstab 2024-04-25T09:32:40.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:40.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_64 /etc/fstab 2024-04-25T09:32:40.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:40.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_65 /etc/fstab 2024-04-25T09:32:40.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:40.433 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_66 /etc/fstab 2024-04-25T09:32:40.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:40.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_67 /etc/fstab 2024-04-25T09:32:40.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:40.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_68 /etc/fstab 2024-04-25T09:32:40.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:40.945 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_69 /etc/fstab 2024-04-25T09:32:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_70 /etc/fstab 2024-04-25T09:32:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:41.288 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_71 /etc/fstab 2024-04-25T09:32:41.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:41.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_72 /etc/fstab 2024-04-25T09:32:41.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:41.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_73 /etc/fstab 2024-04-25T09:32:41.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:41.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_74 /etc/fstab 2024-04-25T09:32:42.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:42.002 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_75 /etc/fstab 2024-04-25T09:32:42.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:42.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_76 /etc/fstab 2024-04-25T09:32:42.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:42.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_77 /etc/fstab 2024-04-25T09:32:42.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:42.521 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_78 /etc/fstab 2024-04-25T09:32:42.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:42.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_79 /etc/fstab 2024-04-25T09:32:42.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:42.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_80 /etc/fstab 2024-04-25T09:32:43.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:43.040 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_81 /etc/fstab 2024-04-25T09:32:43.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:43.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_82 /etc/fstab 2024-04-25T09:32:43.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:43.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_83 /etc/fstab 2024-04-25T09:32:43.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:43.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_84 /etc/fstab 2024-04-25T09:32:43.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:43.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_85 /etc/fstab 2024-04-25T09:32:43.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:43.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_86 /etc/fstab 2024-04-25T09:32:44.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:44.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_87 /etc/fstab 2024-04-25T09:32:44.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:44.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_88 /etc/fstab 2024-04-25T09:32:44.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:44.447 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_89 /etc/fstab 2024-04-25T09:32:44.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:44.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_90 /etc/fstab 2024-04-25T09:32:44.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:44.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_91 /etc/fstab 2024-04-25T09:32:44.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:44.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_92 /etc/fstab 2024-04-25T09:32:45.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:45.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_93 /etc/fstab 2024-04-25T09:32:45.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:45.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_94 /etc/fstab 2024-04-25T09:32:45.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:45.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_95 /etc/fstab 2024-04-25T09:32:45.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:45.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_96 /etc/fstab 2024-04-25T09:32:45.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:45.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_97 /etc/fstab 2024-04-25T09:32:46.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:46.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_98 /etc/fstab 2024-04-25T09:32:46.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:46.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_99 /etc/fstab 2024-04-25T09:32:46.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:32:46.458 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_100 /etc/fstab 2024-04-25T09:32:46.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T09:32:46.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:32:46.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:32:46.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:32:46.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:32:46.626 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T09:32:46.627 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:32:46.627 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:32:46.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:32:46.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:32:46.627 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:32:46.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T09:32:46.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:32:46.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:32:46.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:32:46.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:32:46.701 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:32:47.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:32:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:32:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:32:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T09:32:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T09:32:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T09:32:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:32:47.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:47.125 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:32:47.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836538 2024-04-25T09:32:47.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836538 2024-04-25T09:32:47.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538' 2024-04-25T09:32:47.316 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:47.316 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:32:47.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673008 2024-04-25T09:32:47.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673008 2024-04-25T09:32:47.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-42949673008' 2024-04-25T09:32:47.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:47.480 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:32:47.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509484 2024-04-25T09:32:47.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509484 2024-04-25T09:32:47.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-42949673008 2-64424509484' 2024-04-25T09:32:47.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:47.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:32:47.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345958 2024-04-25T09:32:47.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345958 2024-04-25T09:32:47.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-42949673008 2-64424509484 3-85899345958' 2024-04-25T09:32:47.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:47.794 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:32:47.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182424 2024-04-25T09:32:47.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182424 2024-04-25T09:32:47.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-42949673008 2-64424509484 3-85899345958 4-107374182424' 2024-04-25T09:32:47.948 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:47.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:32:48.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018898 2024-04-25T09:32:48.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018898 2024-04-25T09:32:48.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-42949673008 2-64424509484 3-85899345958 4-107374182424 5-128849018898' 2024-04-25T09:32:48.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:48.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836538 2024-04-25T09:32:48.112 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:48.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:32:48.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836538 2024-04-25T09:32:48.114 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:48.114 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836538 2024-04-25T09:32:48.115 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836538 2024-04-25T09:32:48.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-04-25T09:32:48.115 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:32:48.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836536 -lt 21474836538 2024-04-25T09:32:48.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:32:49.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:32:49.519 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:32:50.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836538 -lt 21474836538 2024-04-25T09:32:50.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:50.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:50.008 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673008 2024-04-25T09:32:50.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:32:50.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673008 2024-04-25T09:32:50.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:50.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673008 2024-04-25T09:32:50.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-04-25T09:32:50.011 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673008 2024-04-25T09:32:50.012 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:32:50.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673009 -lt 42949673008 2024-04-25T09:32:50.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:50.450 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509484 2024-04-25T09:32:50.451 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:50.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:32:50.453 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509484 2024-04-25T09:32:50.454 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:50.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509484 2024-04-25T09:32:50.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2024-04-25T09:32:50.455 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509484 2024-04-25T09:32:50.455 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:32:50.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509485 -lt 64424509484 2024-04-25T09:32:50.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:50.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:50.901 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345958 2024-04-25T09:32:50.901 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:32:50.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345958 2024-04-25T09:32:50.903 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:50.903 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345958 2024-04-25T09:32:50.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345958' 2024-04-25T09:32:50.904 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345958 2024-04-25T09:32:50.904 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:32:51.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345958 -lt 85899345958 2024-04-25T09:32:51.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:51.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182424 2024-04-25T09:32:51.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:51.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:32:51.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182424 2024-04-25T09:32:51.335 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:51.337 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182424 2024-04-25T09:32:51.337 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182424 2024-04-25T09:32:51.338 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182424' 2024-04-25T09:32:51.338 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:32:51.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182424 -lt 107374182424 2024-04-25T09:32:51.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:51.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:51.735 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018898 2024-04-25T09:32:51.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:32:51.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018898 2024-04-25T09:32:51.737 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:51.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018898 2024-04-25T09:32:51.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018898' 2024-04-25T09:32:51.738 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018898 2024-04-25T09:32:51.739 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:32:52.180 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018898 -lt 128849018898 2024-04-25T09:32:52.180 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:32:52.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:32:52.181 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:32:52.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:32:52.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:32:52.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:32:52.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:32:52.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:32:52.719 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:32:52.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:32:52.720 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:32:53.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:32:53.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:32:53.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:32:53.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:32:53.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:32:53.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:32:53.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:32:53.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:84: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 4 5 2024-04-25T09:32:54.624 INFO:tasks.workunit.client.0.smithi102.stderr:set 1.0 pg_upmap mapping to [1,4,5] 2024-04-25T09:32:54.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:87: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:32:56.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:32:56.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:32:56.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:32:56.855 INFO:tasks.workunit.client.0.smithi102.stdout: 2, 2024-04-25T09:32:56.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:93: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back KILL osd.2 2024-04-25T09:32:56.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:32:56.856 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:32:56.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:32:56.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:32:56.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:32:56.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:32:56.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:94: TEST_repeer_on_down_acting_member_coming_back: ceph osd down osd.2 2024-04-25T09:32:57.665 INFO:tasks.workunit.client.0.smithi102.stderr:osd.2 is already down. 2024-04-25T09:32:57.681 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:97: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:32:59.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:32:59.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:32:59.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:32:59.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:103: TEST_repeer_on_down_acting_member_coming_back: activate_osd td/repeer-on-acting-back 2 2024-04-25T09:32:59.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:32:59.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:32:59.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T09:32:59.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:32:59.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repeer-on-acting-back/2 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:32:59.865 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:32:59.866 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:32:59.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repeer-on-acting-back/2 2024-04-25T09:32:59.867 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:32:59.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T09:32:59.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=c5eddd09-73e6-4842-bd28-69ac2fc9d3c1 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.477453/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:32:59.868 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repeer-on-acting-back/2/whoami 2024-04-25T09:32:59.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T09:32:59.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:32:59.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:32:59.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:32:59.898 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:32:59.913+0000 7f8db52db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:59.901 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:32:59.913+0000 7f8db52db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:59.902 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:32:59.917+0000 7f8db52db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:00.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T09:33:00.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:33:00.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:33:00.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:33:00.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:33:00.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:00.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:33:00.295 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:33:00.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:00.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:00.451 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:00.465+0000 7f8db52db7c0 -1 Falling back to public interface 2024-04-25T09:33:00.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:01.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:01.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:01.788 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:33:01.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:33:01.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:01.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:02.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:03.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:03.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:03.241 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:33:03.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:33:03.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:03.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:03.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:04.641 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:04.657+0000 7f8db52db7c0 -1 osd.2 46 log_to_monitors true 2024-04-25T09:33:04.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:04.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:04.787 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:33:04.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:33:04.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:04.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:05.429 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 47 last_clean_interval [15,46) [v2:127.0.0.1:6818/645835207,v1:127.0.0.1:6819/645835207] [v2:127.0.0.1:6820/645835207,v1:127.0.0.1:6821/645835207] exists,up a5080eaa-8b5e-48b8-b649-ac0c77890f4a 2024-04-25T09:33:05.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:33:05.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:33:05.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:33:05.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:104: TEST_repeer_on_down_acting_member_coming_back: wait_for_osd up 2 2024-04-25T09:33:05.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:33:05.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:33:05.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:33:05.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:33:05.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:05.432 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:33:05.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:33:05.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:05.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:33:06.026 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 47 last_clean_interval [15,46) [v2:127.0.0.1:6818/645835207,v1:127.0.0.1:6819/645835207] [v2:127.0.0.1:6820/645835207,v1:127.0.0.1:6821/645835207] exists,up a5080eaa-8b5e-48b8-b649-ac0c77890f4a 2024-04-25T09:33:06.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:33:06.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:33:06.026 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:33:06.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:107: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:33:08.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:33:08.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:33:08.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:33:08.301 INFO:tasks.workunit.client.0.smithi102.stdout: 2 2024-04-25T09:33:08.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T09:33:08.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:33:08.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:33:08.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:33:08.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:33:08.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T09:33:08.303 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:33:08.303 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:33:08.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:33:08.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:33:08.303 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:33:08.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T09:33:08.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:33:08.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:33:08.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:33:08.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:33:08.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:33:08.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:33:08.809 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:33:08.809 INFO:tasks.workunit.client.0.smithi102.stderr:2 2024-04-25T09:33:08.809 INFO:tasks.workunit.client.0.smithi102.stderr:3 2024-04-25T09:33:08.809 INFO:tasks.workunit.client.0.smithi102.stderr:4 2024-04-25T09:33:08.810 INFO:tasks.workunit.client.0.smithi102.stderr:5' 2024-04-25T09:33:08.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:33:08.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:33:08.810 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:33:09.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836544 2024-04-25T09:33:09.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836544 2024-04-25T09:33:09.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544' 2024-04-25T09:33:09.124 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:33:09.124 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:33:09.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673015 2024-04-25T09:33:09.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673015 2024-04-25T09:33:09.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544 1-42949673015' 2024-04-25T09:33:09.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:33:09.344 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:33:09.614 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364803 2024-04-25T09:33:09.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364803 2024-04-25T09:33:09.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544 1-42949673015 2-214748364803' 2024-04-25T09:33:09.615 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:33:09.615 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:33:09.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345964 2024-04-25T09:33:09.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345964 2024-04-25T09:33:09.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544 1-42949673015 2-214748364803 3-85899345964' 2024-04-25T09:33:09.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:33:09.778 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:33:09.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182430 2024-04-25T09:33:09.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182430 2024-04-25T09:33:09.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544 1-42949673015 2-214748364803 3-85899345964 4-107374182430' 2024-04-25T09:33:09.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:33:09.941 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:33:10.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018904 2024-04-25T09:33:10.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018904 2024-04-25T09:33:10.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836544 1-42949673015 2-214748364803 3-85899345964 4-107374182430 5-128849018904' 2024-04-25T09:33:10.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:33:10.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836544 2024-04-25T09:33:10.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:33:10.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:33:10.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836544 2024-04-25T09:33:10.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:33:10.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836544 2024-04-25T09:33:10.111 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836544 2024-04-25T09:33:10.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836544' 2024-04-25T09:33:10.111 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:33:10.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836542 -lt 21474836544 2024-04-25T09:33:10.536 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:33:11.538 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:33:11.538 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:33:11.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836545 -lt 21474836544 2024-04-25T09:33:11.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:33:11.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673015 2024-04-25T09:33:11.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:33:11.974 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:33:11.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673015 2024-04-25T09:33:11.975 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:33:11.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673015 2024-04-25T09:33:11.976 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 42949673015 2024-04-25T09:33:11.977 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673015' 2024-04-25T09:33:11.977 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:33:12.498 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673015 -lt 42949673015 2024-04-25T09:33:12.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:33:12.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-214748364803 2024-04-25T09:33:12.499 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:33:12.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:33:12.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-214748364803 2024-04-25T09:33:12.502 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:33:12.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364803 2024-04-25T09:33:12.503 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 214748364803 2024-04-25T09:33:12.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-04-25T09:33:12.504 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:33:13.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364804 -lt 214748364803 2024-04-25T09:33:13.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:33:13.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345964 2024-04-25T09:33:13.104 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:33:13.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:33:13.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345964 2024-04-25T09:33:13.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:33:13.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345964 2024-04-25T09:33:13.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345964' 2024-04-25T09:33:13.108 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.3 seq 85899345964 2024-04-25T09:33:13.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:33:13.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345965 -lt 85899345964 2024-04-25T09:33:13.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:33:13.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182430 2024-04-25T09:33:13.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:33:13.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:33:13.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182430 2024-04-25T09:33:13.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:33:13.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182430 2024-04-25T09:33:13.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182430' 2024-04-25T09:33:13.564 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.4 seq 107374182430 2024-04-25T09:33:13.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:33:13.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182431 -lt 107374182430 2024-04-25T09:33:13.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:33:13.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018904 2024-04-25T09:33:13.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:33:13.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:33:13.997 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018904 2024-04-25T09:33:13.998 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:33:13.998 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018904 2024-04-25T09:33:13.999 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.5 seq 128849018904 2024-04-25T09:33:13.999 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018904' 2024-04-25T09:33:13.999 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:33:14.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018904 -lt 128849018904 2024-04-25T09:33:14.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:33:14.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:33:14.513 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:33:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:33:15.294 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:33:15.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:33:15.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:33:15.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:33:15.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:33:15.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:33:15.294 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:33:15.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:33:15.723 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:33:15.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:33:15.724 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:33:16.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:33:16.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T09:33:16.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:33:16.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T09:33:16.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:33:16.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:33:16.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:33:16.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:33:16.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:33:16.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:33:16.532 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:33:16.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:33:16.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:33:16.533 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:33:17.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:33:17.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:33:17.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:33:17.010 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:33:17.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:33:17.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:33:17.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:33:17.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:33:17.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:33:17.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:33:17.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:33:17.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:33:17.617 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:33:18.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2393 2024-04-25T09:33:18.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2393 '!=' null 2024-04-25T09:33:18.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:33:18.155 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:33:18.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:33:18.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:33:18.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:33:18.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:33:18.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:33:18.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:33:18.257 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:33:18.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:33:18.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:33:18.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:33:18.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:33:18.658 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:33:18.659 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:33:19.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:33:19.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:33:19.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:33:19.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:33:19.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:33:19.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:33:19.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:33:19.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:33:19.201 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:33:19.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3088 2024-04-25T09:33:19.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3088 '!=' null 2024-04-25T09:33:19.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:33:19.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:33:19.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:33:19.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:33:19.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:33:19.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:33:19.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:33:19.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:33:19.863 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:33:19.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:33:19.864 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:33:20.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:33:20.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:33:20.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:33:20.285 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:33:20.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:33:20.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:33:20.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:33:20.843 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: find td/repeer-on-acting-back -name '*osd*log' 2024-04-25T09:33:20.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: grep -q 'Active: got notify from previous acting member.*, requesting pg_temp change' td/repeer-on-acting-back/osd.5.log td/repeer-on-acting-back/osd.3.log td/repeer-on-acting-back/osd.4.log td/repeer-on-acting-back/osd.2.log td/repeer-on-acting-back/osd.1.log td/repeer-on-acting-back/osd.0.log 2024-04-25T09:33:20.854 INFO:tasks.workunit.client.0.smithi102.stdout:success 2024-04-25T09:33:20.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:119: TEST_repeer_on_down_acting_member_coming_back: echo success 2024-04-25T09:33:20.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:121: TEST_repeer_on_down_acting_member_coming_back: delete_pool test 2024-04-25T09:33:20.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T09:33:20.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T09:33:21.484 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' does not exist 2024-04-25T09:33:21.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:122: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back 2024-04-25T09:33:21.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:33:21.501 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:33:21.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:33:21.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:33:21.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:33:22.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:33:22.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:42: run: teardown td/repeer-on-acting-back 2024-04-25T09:33:22.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:33:22.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:33:22.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:33:22.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:33:22.816 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:33:22.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:33:22.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:33:22.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:33:22.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:33:22.826 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:33:22.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:33:22.827 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:33:22.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:33:22.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:33:22.828 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:33:22.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:33:22.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:33:22.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:33:22.830 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:33:22.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:33:22.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:33:22.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:33:22.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:33:22.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:33:22.871 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:33:22.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.477453 2024-04-25T09:33:22.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:33:22.872 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:33:22.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:33:22.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/repeer-on-acting-back 0 2024-04-25T09:33:22.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:33:22.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:33:22.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:33:22.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:33:22.873 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:33:22.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:33:22.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:33:22.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:33:22.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:33:22.874 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:33:22.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:33:22.875 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:33:22.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:33:22.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:33:22.877 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:33:22.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:33:22.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:33:22.878 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:33:22.878 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:33:22.879 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:33:22.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:33:22.880 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:33:22.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:33:22.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:33:22.881 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.477453 2024-04-25T09:33:22.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.477453 2024-04-25T09:33:22.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:33:22.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:33:22.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:33:22.883 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:33:22.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:33:22.894 INFO:tasks.workunit:Running workunit osd/repro_long_log.sh... 2024-04-25T09:33:22.894 DEBUG:teuthology.orchestra.run.smithi102:workunit test osd/repro_long_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh 2024-04-25T09:33:22.946 INFO:tasks.workunit.client.0.smithi102.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:33:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:33:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:33:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:33:22.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/repro-long-log 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:23: run: local dir=td/repro-long-log 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:24: run: shift 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: export CEPH_MON=127.0.0.1:7100 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: CEPH_MON=127.0.0.1:7100 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:27: run: export CEPH_ARGS 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: uuidgen 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: CEPH_ARGS+='--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none ' 2024-04-25T09:33:22.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-04-25T09:33:22.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: set 2024-04-25T09:33:22.954 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: local 'funcs=TEST_repro_long_log1 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_repro_long_log2 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_trim_max_entries 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:TEST_trim_max_entries_with_dups' 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:33:22.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:33:22.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:33:22.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:33:22.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:33:22.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:33:22.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:33:22.958 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:33:22.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:33:22.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:33:22.960 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:33:22.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:33:22.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:33:22.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:33:22.961 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:33:22.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:33:22.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:33:22.962 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:33:22.963 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:33:22.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:33:22.964 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:33:22.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493206 2024-04-25T09:33:22.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:33:22.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:33:22.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:33:22.965 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:33:22.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:33:22.966 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:33:22.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.493206 2024-04-25T09:33:22.968 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:33:22.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:33:22.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:33:22.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:33:22.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:33:22.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log1 td/repro-long-log 2024-04-25T09:33:22.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:97: TEST_repro_long_log1: local dir=td/repro-long-log 2024-04-25T09:33:22.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:99: TEST_repro_long_log1: setup_log_test td/repro-long-log 2024-04-25T09:33:22.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:33:22.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:33:22.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:33:22.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:33:22.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:33:22.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:33:22.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:33:22.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:33:22.969 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:33:23.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:33:23.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:33:23.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:33:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:33:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:33:23.408 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:33:23.408 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:33:23.409 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:33:23.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:33:23.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:33:23.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:33:23.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:33:23.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:33:23.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:33:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:33:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:33:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:33:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:33:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:33:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:33:23.621 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493206/ceph-mon.a.asok 2024-04-25T09:33:23.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:33:23.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493206/ceph-mon.a.asok config get fsid 2024-04-25T09:33:23.730 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:33:23.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:33:23.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:33:23.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:33:23.731 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:33:23.732 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:33:23.732 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:33:23.732 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:33:23.733 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:33:23.733 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:33:23.733 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:33:23.733 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493206/ceph-mon.a.asok 2024-04-25T09:33:23.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:33:23.733 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493206/ceph-mon.a.asok config get mon_host 2024-04-25T09:33:23.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:33:23.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:33:23.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:33:23.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:33:23.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:33:23.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:33:23.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:33:24.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:33:24.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:33:24.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:33:24.332 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:33:24.332 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:33:24.332 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:33:24.332 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:33:24.333 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:33:24.334 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:33:24.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:33:24.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:33:24.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:33:24.370 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:33:24.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:33:24.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:33:24.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:33:24.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:33:24.372 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:33:24.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b3e8d724-ec7e-4eec-9046-f5598b969398 2024-04-25T09:33:24.372 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 b3e8d724-ec7e-4eec-9046-f5598b969398 2024-04-25T09:33:24.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b3e8d724-ec7e-4eec-9046-f5598b969398' 2024-04-25T09:33:24.373 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:33:24.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBkIypmp+8ZGBAAIw2ptsQ5bmsWlVHpKfAmQA== 2024-04-25T09:33:24.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBkIypmp+8ZGBAAIw2ptsQ5bmsWlVHpKfAmQA=="}' 2024-04-25T09:33:24.387 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b3e8d724-ec7e-4eec-9046-f5598b969398 -i td/repro-long-log/0/new.json 2024-04-25T09:33:24.672 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:33:24.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:33:24.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkIypmp+8ZGBAAIw2ptsQ5bmsWlVHpKfAmQA== --osd-uuid b3e8d724-ec7e-4eec-9046-f5598b969398 2024-04-25T09:33:24.724 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:24.736+0000 7f69f028a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:24.740 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:24.752+0000 7f69f028a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:24.741 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:24.756+0000 7f69f028a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:24.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:24.756+0000 7f69f028a7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:33:24.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:24.756+0000 7f69f028a7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:33:24.742 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:24.756+0000 7f69f028a7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:33:24.780 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:24.792+0000 7f69f028a7c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:33:30.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:33:30.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:33:30.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:33:30.661 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:33:30.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:33:31.284 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:33:31.284 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:33:31.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:33:31.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:33:31.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:33:31.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:33:31.314 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:31.328+0000 7fdd6a8087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:31.321 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:31.336+0000 7fdd6a8087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:31.322 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:31.336+0000 7fdd6a8087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:31.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:33:31.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:33:31.709 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:33:31.709 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:33:31.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:33:31.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:33:31.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:31.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:33:31.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:31.710 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:31.878 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:31.888+0000 7fdd6a8087c0 -1 Falling back to public interface 2024-04-25T09:33:32.121 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:33.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:33.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:33.122 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:33:33.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:33:33.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:33.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:33.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:34.337 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:34.348+0000 7fdd6a8087c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:33:34.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:34.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:34.547 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:33:34.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:33:34.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:34.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:35.030 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:36.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:36.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:36.032 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:33:36.032 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:33:36.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:36.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:36.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:37.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:37.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:33:37.481 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:33:37.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:37.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:37.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:38.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:38.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:38.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:33:38.886 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:33:38.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:38.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:39.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:40.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:40.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:40.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:33:40.333 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:33:40.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:40.333 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:40.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:41.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:41.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:41.801 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:33:41.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:33:41.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:41.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:42.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:43.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:43.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:43.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:33:43.357 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:33:43.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:43.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:43.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:44.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:44.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:44.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:33:44.809 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:33:44.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:44.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:45.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:33:46.547 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:33:46.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:46.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:47.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:48.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:48.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:48.061 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:33:48.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:33:48.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:48.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:48.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:49.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:49.503 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:49.503 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:33:49.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:33:49.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:49.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:50.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:51.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:51.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:51.292 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:33:51.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:33:51.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:51.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:52.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:53.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:53.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:53.454 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:33:53.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:33:53.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:53.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:54.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:55.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:55.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:55.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:33:55.749 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:33:55.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:55.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:56.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:33:57.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:33:57.210 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:33:57.211 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:33:57.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:33:57.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:33:57.211 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:33:57.663 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1250720241,v1:127.0.0.1:6803/1250720241] [v2:127.0.0.1:6804/1250720241,v1:127.0.0.1:6805/1250720241] exists,up b3e8d724-ec7e-4eec-9046-f5598b969398 2024-04-25T09:33:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:33:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:33:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:33:57.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:33:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:33:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:33:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:33:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:33:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:33:57.669 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:33:57.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:33:57.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:33:57.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:33:57.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:33:57.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:33:57.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:33:57.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:33:57.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:33:57.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:33:57.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:33:57.671 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:33:57.672 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:33:57.672 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:33:57.672 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:33:57.672 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:33:57.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:33:57.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=49ae16e5-a267-47d2-bb55-6154c993f0f7 2024-04-25T09:33:57.673 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 49ae16e5-a267-47d2-bb55-6154c993f0f7' 2024-04-25T09:33:57.674 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 49ae16e5-a267-47d2-bb55-6154c993f0f7 2024-04-25T09:33:57.674 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:33:57.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFIypmsksQKhAA/+UaI60sEOwsc8Qtwl5zMQ== 2024-04-25T09:33:57.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFIypmsksQKhAA/+UaI60sEOwsc8Qtwl5zMQ=="}' 2024-04-25T09:33:57.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 49ae16e5-a267-47d2-bb55-6154c993f0f7 -i td/repro-long-log/1/new.json 2024-04-25T09:33:58.184 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:33:58.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:33:58.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFIypmsksQKhAA/+UaI60sEOwsc8Qtwl5zMQ== --osd-uuid 49ae16e5-a267-47d2-bb55-6154c993f0f7 2024-04-25T09:33:58.232 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:58.247+0000 7f925e1a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:58.269 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:58.287+0000 7f925e1a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:58.270 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:58.287+0000 7f925e1a77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:33:58.271 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:58.287+0000 7f925e1a77c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:33:58.271 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:58.287+0000 7f925e1a77c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:33:58.271 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:58.287+0000 7f925e1a77c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:33:58.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:33:58.331+0000 7f925e1a77c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:34:04.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:34:04.158 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:34:04.160 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:34:04.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:34:04.160 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:34:04.715 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:34:04.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:34:04.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:34:04.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:34:04.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:34:04.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:34:04.744 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:04.759+0000 7f7d16f567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:04.773 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:04.787+0000 7f7d16f567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:04.774 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:04.791+0000 7f7d16f567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:05.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:34:05.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:34:05.177 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:34:05.177 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:34:05.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:34:05.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:34:05.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:05.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:34:05.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:05.178 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:05.347 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:05.363+0000 7f7d16f567c0 -1 Falling back to public interface 2024-04-25T09:34:05.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:06.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:06.696 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:06.696 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:34:06.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:34:06.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:06.697 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:07.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:07.975 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:07.991+0000 7f7d16f567c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:34:08.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:08.129 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:08.130 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:34:08.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:34:08.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:08.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:08.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:09.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:09.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:09.640 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:34:09.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:34:09.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:09.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:10.138 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:11.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:11.140 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:11.140 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:34:11.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:34:11.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:11.141 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:11.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:12.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:12.694 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:12.695 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:34:12.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:34:12.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:12.695 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:13.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:14.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:14.145 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:14.145 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:34:14.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:34:14.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:14.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:14.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:15.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:15.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:15.547 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:34:15.547 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:34:15.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:15.548 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:16.073 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:17.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:17.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:17.076 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:34:17.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:34:17.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:17.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:17.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:18.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:18.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:18.517 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:34:18.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:34:18.517 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:18.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:18.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:19.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:19.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:19.955 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:34:19.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:34:19.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:19.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:20.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:21.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:21.834 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:34:21.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:34:21.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:21.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:22.573 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:23.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:23.575 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:34:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:34:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:23.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:24.317 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:25.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:25.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:25.319 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:34:25.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:34:25.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:25.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:26.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:27.152 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:27.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:27.153 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:34:27.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:34:27.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:27.154 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:28.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:29.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:29.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:29.053 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:34:29.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:34:29.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:29.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:29.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:30.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:30.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:30.989 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:34:30.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:34:30.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:30.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:34:31.418 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/979175179,v1:127.0.0.1:6811/979175179] [v2:127.0.0.1:6812/979175179,v1:127.0.0.1:6813/979175179] exists,up 49ae16e5-a267-47d2-bb55-6154c993f0f7 2024-04-25T09:34:31.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:34:31.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:34:31.422 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:34:31.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:34:31.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:34:31.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:34:31.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:34:31.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:34:31.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:34:31.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:34:31.424 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:34:31.426 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:34:31.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c57ebfd8-5efc-49e1-92d6-c69150f6ad19 2024-04-25T09:34:31.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c57ebfd8-5efc-49e1-92d6-c69150f6ad19' 2024-04-25T09:34:31.427 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 c57ebfd8-5efc-49e1-92d6-c69150f6ad19 2024-04-25T09:34:31.428 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:34:31.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnIypmyT6yGxAAMUElsbC4Ig51u500E+lBoQ== 2024-04-25T09:34:31.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnIypmyT6yGxAAMUElsbC4Ig51u500E+lBoQ=="}' 2024-04-25T09:34:31.450 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c57ebfd8-5efc-49e1-92d6-c69150f6ad19 -i td/repro-long-log/2/new.json 2024-04-25T09:34:31.894 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:34:31.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:34:31.911 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnIypmyT6yGxAAMUElsbC4Ig51u500E+lBoQ== --osd-uuid c57ebfd8-5efc-49e1-92d6-c69150f6ad19 2024-04-25T09:34:31.943 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:31.959+0000 7fecce9607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:31.952 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:31.967+0000 7fecce9607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:31.953 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:31.967+0000 7fecce9607c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:31.953 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:31.971+0000 7fecce9607c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:34:31.953 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:31.971+0000 7fecce9607c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:34:31.954 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:31.971+0000 7fecce9607c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:34:32.036 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:32.051+0000 7fecce9607c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:34:37.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:34:37.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:34:37.847 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T09:34:37.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:34:37.848 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:34:38.402 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:34:38.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:34:38.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:34:38.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:34:38.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:34:38.404 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:34:38.432 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:38.447+0000 7f5b17da27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:38.458 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:38.475+0000 7f5b17da27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:38.459 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:38.475+0000 7f5b17da27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:38.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:34:38.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:34:38.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:34:38.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:34:38.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:34:38.808 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:38.808 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:34:38.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:34:38.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:38.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:39.006 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:39.023+0000 7f5b17da27c0 -1 Falling back to public interface 2024-04-25T09:34:39.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:40.290 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:40.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:40.291 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:34:40.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:34:40.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:40.292 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:40.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:41.626 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:34:41.643+0000 7f5b17da27c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:34:41.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:41.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:41.701 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:34:41.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:34:41.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:41.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:42.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:43.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:43.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:43.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:34:43.304 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:34:43.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:43.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:43.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:44.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:44.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:44.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:34:44.886 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:34:44.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:44.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:45.480 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:46.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:46.482 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:34:46.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:46.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:34:46.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:46.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:46.896 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:47.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:47.898 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:47.898 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:34:47.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:34:47.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:47.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:48.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:49.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:49.419 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:34:49.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:34:49.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:49.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:49.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:50.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:50.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:50.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:34:50.954 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:34:50.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:50.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:51.479 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:52.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:52.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:52.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:34:52.481 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:34:52.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:52.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:52.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:53.933 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:53.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:53.934 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:34:53.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:34:53.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:53.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:54.482 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:55.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:55.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:55.483 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:34:55.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:34:55.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:55.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:56.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:57.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:57.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:57.475 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:34:57.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:34:57.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:57.475 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:58.187 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:59.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:59.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:59.189 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:34:59.190 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:34:59.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:59.190 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:34:59.954 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:00.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:00.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:00.956 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:35:00.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:35:00.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:00.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:35:02.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:03.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:03.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:03.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:35:03.276 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:35:03.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:03.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:35:03.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:04.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:04.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:04.996 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:35:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:35:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:04.996 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:35:05.445 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/716143344,v1:127.0.0.1:6819/716143344] [v2:127.0.0.1:6820/716143344,v1:127.0.0.1:6821/716143344] exists,up c57ebfd8-5efc-49e1-92d6-c69150f6ad19 2024-04-25T09:35:05.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:35:05.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:35:05.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:35:05.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:35:05.963 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T09:35:05.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:35:05.981 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:35:06.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:35:06.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:35:06.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:35:06.606 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:35:06.633 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:35:06.659 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:35:06.672 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:35:06.808 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:35:06.838 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:35:06.864 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:35:06.877 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:35:07.157 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:35:07.194 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:35:07.227 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:35:07.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:35:07.396 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:35:07.423 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:35:07.452 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:35:07.467 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:35:07.610 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:35:07.641 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:35:07.673 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:35:07.688 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:35:07.689 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:35:07.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:07.690 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:07.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:07.993 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:08.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:08.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:08.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:08.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:08.344 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:08.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:08.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:08.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:08.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:08.687 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:08.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:08.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:08.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:08.947 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:09.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:09.064 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:09.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:09.182 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:09.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:09.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:09.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:09.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:09.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:09.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:09.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:09.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:09.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:09.794 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:09.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:09.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:10.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:10.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:10.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:10.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:10.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:35:10.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:35:10.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:35:10.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:35:10.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:35:10.380 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836501 2024-04-25T09:35:10.404 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672971 2024-04-25T09:35:10.428 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509450 2024-04-25T09:35:10.443 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:35:13.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:13.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:35:13.608 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:35:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:35:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:35:13.609 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:13.763 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:35:13.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:13.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:35:13.917 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:35:13.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:13.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:35:13.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:35:14.075 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:35:14.075 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:35:14.199 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:35:14.200 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:14.201 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:14.295 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:14.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:14.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:14.387 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:14.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:14.390 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:14.479 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:14.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:14.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:14.597 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:14.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:14.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:14.689 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:14.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:14.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:14.789 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:14.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:14.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:14.890 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:14.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:14.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:15.083 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:15.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:15.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:15.216 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:15.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:15.219 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:15.342 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:15.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:15.345 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:15.442 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:15.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:15.446 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:15.585 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:15.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:15.589 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:15.711 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:15.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:15.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:15.812 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:15.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:15.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:15.904 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:15.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:15.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:16.004 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:16.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:16.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:16.105 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:16.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:16.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:16.239 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:16.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:16.339 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:16.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:35:16.343 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:35:16.473 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:35:16.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:35:16.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:35:16.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:35:16.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:35:16.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:35:16.625 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836504 2024-04-25T09:35:16.650 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672975 2024-04-25T09:35:16.674 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509453 2024-04-25T09:35:16.689 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:35:19.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:19.691 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:35:19.856 INFO:tasks.workunit.client.0.smithi102.stdout:21 2024-04-25T09:35:19.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:19.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:35:19.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:35:20.015 INFO:tasks.workunit.client.0.smithi102.stdout:21 2024-04-25T09:35:20.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:20.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:35:20.169 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:35:20.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:20.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:35:20.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:35:20.327 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:35:20.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:101: TEST_repro_long_log1: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:35:20.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:102: TEST_repro_long_log1: test_log_size 1.0 22 2024-04-25T09:35:20.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:35:20.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=22 2024-04-25T09:35:20.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:35:20.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:35:20.569 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836507 2024-04-25T09:35:20.593 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672977 2024-04-25T09:35:20.617 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509456 2024-04-25T09:35:20.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:35:23.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:23.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:35:23.799 INFO:tasks.workunit.client.0.smithi102.stdout:22 2024-04-25T09:35:23.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:23.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:35:23.800 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 22 2024-04-25T09:35:23.956 INFO:tasks.workunit.client.0.smithi102.stdout:22 2024-04-25T09:35:23.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:23.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:35:24.108 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:35:24.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:35:24.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:35:24.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:35:24.264 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:35:24.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:35:24.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:35:24.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:35:24.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:35:24.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:35:24.266 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:35:24.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:35:24.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:35:24.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:35:24.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:35:24.575 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:35:24.576 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:35:24.577 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:35:24.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:35:24.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:35:24.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:35:24.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:35:24.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:35:24.581 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:35:24.581 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:35:24.582 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:35:24.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:35:24.584 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:35:24.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:35:24.604 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:35:24.605 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:35:24.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493206 2024-04-25T09:35:24.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:35:24.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:35:24.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:35:24.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:35:24.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:35:24.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:35:24.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:35:24.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:35:24.607 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:35:24.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:35:24.607 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:35:24.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:35:24.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:35:24.608 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:35:24.610 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:35:24.610 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:35:24.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:35:24.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:35:24.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:35:24.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:35:24.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:35:24.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:35:24.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:35:24.619 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:35:24.619 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:35:24.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:35:24.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:35:24.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:35:24.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:35:24.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:35:24.620 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:35:24.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493206 2024-04-25T09:35:24.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:35:24.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:35:24.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:35:24.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:35:24.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:35:24.621 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:35:24.621 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.493206 2024-04-25T09:35:24.622 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:35:24.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:35:24.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:35:24.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:35:24.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log2 td/repro-long-log 2024-04-25T09:35:24.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:107: TEST_repro_long_log2: local dir=td/repro-long-log 2024-04-25T09:35:24.624 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:109: TEST_repro_long_log2: setup_log_test td/repro-long-log 2024-04-25T09:35:24.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:35:24.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:35:24.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:35:24.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:35:24.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:35:24.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:35:24.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:35:24.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:35:24.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:35:25.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:35:25.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:35:25.088 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:35:25.088 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:35:25.089 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:35:25.089 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:35:25.089 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:35:25.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:35:25.289 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:35:25.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:35:25.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:35:25.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:35:25.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:35:25.291 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:35:25.291 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:35:25.291 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:35:25.291 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:35:25.292 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:35:25.292 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:35:25.292 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:35:25.292 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493206/ceph-mon.a.asok 2024-04-25T09:35:25.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:35:25.292 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493206/ceph-mon.a.asok config get fsid 2024-04-25T09:35:25.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:35:25.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:35:25.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:35:25.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:35:25.405 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:35:25.406 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:35:25.406 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:35:25.406 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:35:25.407 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:35:25.407 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:35:25.407 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:35:25.407 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493206/ceph-mon.a.asok 2024-04-25T09:35:25.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:35:25.407 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493206/ceph-mon.a.asok config get mon_host 2024-04-25T09:35:25.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:35:25.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:35:25.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:35:25.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:35:25.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:35:25.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:35:25.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:35:26.019 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:35:26.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:35:26.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:35:26.020 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:35:26.020 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:35:26.020 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:35:26.020 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:35:26.021 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:35:26.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:35:26.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:35:26.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:35:26.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:35:26.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:35:26.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:35:26.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:35:26.058 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:35:26.059 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:35:26.060 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:35:26.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=460b47d8-268b-4545-8f26-7df30c10bb2c 2024-04-25T09:35:26.061 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 460b47d8-268b-4545-8f26-7df30c10bb2c 2024-04-25T09:35:26.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 460b47d8-268b-4545-8f26-7df30c10bb2c' 2024-04-25T09:35:26.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:35:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDeIypmYVStBRAAOrbsezfTToC4Vm/ZIOiycg== 2024-04-25T09:35:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDeIypmYVStBRAAOrbsezfTToC4Vm/ZIOiycg=="}' 2024-04-25T09:35:26.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 460b47d8-268b-4545-8f26-7df30c10bb2c -i td/repro-long-log/0/new.json 2024-04-25T09:35:26.377 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:35:26.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:35:26.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDeIypmYVStBRAAOrbsezfTToC4Vm/ZIOiycg== --osd-uuid 460b47d8-268b-4545-8f26-7df30c10bb2c 2024-04-25T09:35:26.419 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:26.434+0000 7f774d0007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:26.438 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:26.454+0000 7f774d0007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:26.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:26.454+0000 7f774d0007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:26.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:26.454+0000 7f774d0007c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:35:26.439 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:26.454+0000 7f774d0007c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:35:26.440 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:26.454+0000 7f774d0007c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:35:26.478 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:26.494+0000 7f774d0007c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:35:32.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:35:32.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:35:32.420 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:35:32.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:35:32.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:35:32.996 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:35:32.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:35:32.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:35:32.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:35:32.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:35:32.997 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:35:33.026 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:33.041+0000 7f5f3414c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:33.028 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:33.041+0000 7f5f3414c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:33.029 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:33.041+0000 7f5f3414c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:33.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:35:33.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:35:33.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:35:33.430 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:35:33.430 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:35:33.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:35:33.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:33.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:35:33.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:33.431 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:33.603 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:33.617+0000 7f5f3414c7c0 -1 Falling back to public interface 2024-04-25T09:35:33.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:34.844 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:34.845 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:35:34.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:34.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:35:34.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:34.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:35.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:36.092 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:36.105+0000 7f5f3414c7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:35:36.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:36.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:36.245 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:35:36.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:35:36.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:36.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:36.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:37.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:37.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:37.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:35:37.824 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:35:37.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:37.825 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:38.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:39.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:39.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:39.245 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:35:39.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:35:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:39.246 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:39.829 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:40.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:40.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:40.831 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:35:40.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:35:40.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:40.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:41.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:42.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:42.238 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:35:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:35:42.238 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:42.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:42.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:43.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:43.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:43.826 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:35:43.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:35:43.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:43.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:44.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:45.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:45.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:45.271 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:35:45.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:35:45.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:45.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:45.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:46.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:46.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:46.887 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:35:46.887 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:35:46.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:46.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:47.481 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:48.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:48.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:48.484 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:35:48.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:35:48.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:48.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:49.010 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:50.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:50.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:50.012 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:35:50.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:35:50.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:50.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:50.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:51.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:51.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:51.416 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:35:51.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:35:51.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:51.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:52.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:53.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:53.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:53.092 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:35:53.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:35:53.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:53.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:54.031 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:55.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:55.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:55.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:35:55.033 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:35:55.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:55.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:55.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:56.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:35:56.905 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:35:56.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:56.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:57.592 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:58.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:58.594 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:58.594 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:35:58.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:35:58.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:58.595 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:59.044 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3126014762,v1:127.0.0.1:6803/3126014762] [v2:127.0.0.1:6804/3126014762,v1:127.0.0.1:6805/3126014762] exists,up 460b47d8-268b-4545-8f26-7df30c10bb2c 2024-04-25T09:35:59.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:35:59.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:35:59.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:35:59.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:35:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:35:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:35:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:35:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:35:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:35:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:35:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:35:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:35:59.050 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:35:59.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aad66ef2-8259-4b99-8047-20faf315b1a3 2024-04-25T09:35:59.051 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 aad66ef2-8259-4b99-8047-20faf315b1a3 2024-04-25T09:35:59.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 aad66ef2-8259-4b99-8047-20faf315b1a3' 2024-04-25T09:35:59.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:35:59.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/Iypm+NAMBRAARxMsyQ2HNtFIbGrTXbaa+g== 2024-04-25T09:35:59.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/Iypm+NAMBRAARxMsyQ2HNtFIbGrTXbaa+g=="}' 2024-04-25T09:35:59.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aad66ef2-8259-4b99-8047-20faf315b1a3 -i td/repro-long-log/1/new.json 2024-04-25T09:35:59.516 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:35:59.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:35:59.533 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/Iypm+NAMBRAARxMsyQ2HNtFIbGrTXbaa+g== --osd-uuid aad66ef2-8259-4b99-8047-20faf315b1a3 2024-04-25T09:35:59.557 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:59.569+0000 7f8bc4ce77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:59.561 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:59.573+0000 7f8bc4ce77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:59.562 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:59.577+0000 7f8bc4ce77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:59.563 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:59.577+0000 7f8bc4ce77c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:35:59.563 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:59.577+0000 7f8bc4ce77c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:35:59.563 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:59.577+0000 7f8bc4ce77c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:35:59.600 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:35:59.613+0000 7f8bc4ce77c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:36:05.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:36:05.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:36:05.521 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:36:05.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:36:05.522 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:36:06.070 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:36:06.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:36:06.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:36:06.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:36:06.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:36:06.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:36:06.101 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:06.113+0000 7f34044577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:06.107 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:06.121+0000 7f34044577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:06.108 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:06.121+0000 7f34044577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:06.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:36:06.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:36:06.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:36:06.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:36:06.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:36:06.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:06.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:36:06.484 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:36:06.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:06.484 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:06.676 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:06.689+0000 7f34044577c0 -1 Falling back to public interface 2024-04-25T09:36:06.965 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:07.966 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:07.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:07.967 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:36:07.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:36:07.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:07.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:08.432 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:09.305 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:09.317+0000 7f34044577c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:36:09.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:09.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:09.434 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:36:09.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:36:09.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:09.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:09.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:10.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:10.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:10.960 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:36:10.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:36:10.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:10.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:11.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:12.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:12.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:12.495 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:36:12.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:36:12.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:12.495 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:12.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:13.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:13.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:13.918 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:36:13.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:36:13.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:13.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:14.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:15.356 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:15.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:15.357 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:36:15.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:36:15.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:15.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:15.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:16.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:16.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:16.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:36:16.824 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:36:16.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:16.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:17.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:18.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:18.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:18.384 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:36:18.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:36:18.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:18.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:18.790 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:19.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:19.792 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:19.792 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:36:19.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:36:19.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:19.793 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:20.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:21.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:21.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:21.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:36:21.402 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:36:21.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:21.402 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:22.220 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:23.222 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:23.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:23.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:36:23.223 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:36:23.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:23.223 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:24.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:25.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:25.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:25.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:36:25.197 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:36:25.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:25.197 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:26.257 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:27.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:27.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:27.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:36:27.260 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:36:27.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:27.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:27.906 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:28.907 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:28.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:28.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:36:28.908 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:36:28.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:28.908 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:29.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:30.666 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:30.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:30.667 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:36:30.667 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:36:30.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:30.668 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:31.202 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:32.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:32.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:32.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:36:32.204 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:36:32.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:32.205 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:32.628 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2610888914,v1:127.0.0.1:6811/2610888914] [v2:127.0.0.1:6812/2610888914,v1:127.0.0.1:6813/2610888914] exists,up aad66ef2-8259-4b99-8047-20faf315b1a3 2024-04-25T09:36:32.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:36:32.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:36:32.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:36:32.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:36:32.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:36:32.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:36:32.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:36:32.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:36:32.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:36:32.632 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:36:32.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:36:32.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:36:32.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:36:32.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:36:32.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:36:32.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:36:32.634 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:36:32.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e0f0c195-3b3d-4b59-b9a4-f8ddcb87644b 2024-04-25T09:36:32.635 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 e0f0c195-3b3d-4b59-b9a4-f8ddcb87644b 2024-04-25T09:36:32.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e0f0c195-3b3d-4b59-b9a4-f8ddcb87644b' 2024-04-25T09:36:32.635 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:36:32.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAgJCpmkefUJxAAhpSxyw2TGHGmLfS2A3Ptkg== 2024-04-25T09:36:32.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAgJCpmkefUJxAAhpSxyw2TGHGmLfS2A3Ptkg=="}' 2024-04-25T09:36:32.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e0f0c195-3b3d-4b59-b9a4-f8ddcb87644b -i td/repro-long-log/2/new.json 2024-04-25T09:36:33.127 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:36:33.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:36:33.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgJCpmkefUJxAAhpSxyw2TGHGmLfS2A3Ptkg== --osd-uuid e0f0c195-3b3d-4b59-b9a4-f8ddcb87644b 2024-04-25T09:36:33.175 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:33.184+0000 7f39017c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:33.177 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:33.188+0000 7f39017c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:33.178 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:33.188+0000 7f39017c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:33.188+0000 7f39017c57c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:36:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:33.188+0000 7f39017c57c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:36:33.179 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:33.188+0000 7f39017c57c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:36:33.219 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:33.232+0000 7f39017c57c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:36:39.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:36:39.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:36:39.050 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T09:36:39.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:36:39.051 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:36:39.648 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:36:39.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:36:39.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:36:39.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:36:39.649 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:36:39.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:36:39.678 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:39.688+0000 7ff5885297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:39.710 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:39.720+0000 7ff5885297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:39.712 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:39.724+0000 7ff5885297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:40.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:36:40.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:36:40.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:36:40.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:36:40.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:36:40.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:40.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:36:40.069 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:36:40.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:40.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:40.286 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:40.296+0000 7ff5885297c0 -1 Falling back to public interface 2024-04-25T09:36:40.510 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:41.511 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:41.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:41.512 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:36:41.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:36:41.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:41.512 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:41.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:42.862 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:36:42.872+0000 7ff5885297c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:36:42.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:42.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:42.959 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:36:42.959 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:36:42.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:42.960 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:43.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:44.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:44.545 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:44.545 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:36:44.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:36:44.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:44.546 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:45.050 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:46.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:46.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:46.053 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:36:46.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:36:46.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:46.054 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:46.486 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:47.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:47.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:47.489 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:36:47.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:36:47.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:47.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:48.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:49.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:49.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:49.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:36:49.016 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:36:49.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:49.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:49.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:50.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:50.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:50.605 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:36:50.605 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:36:50.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:50.606 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:51.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:52.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:52.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:52.071 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:36:52.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:36:52.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:52.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:52.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:53.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:53.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:53.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:36:53.556 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:36:53.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:53.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:54.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:55.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:55.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:55.071 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:36:55.071 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:36:55.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:55.072 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:55.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:56.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:56.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:56.574 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:36:56.574 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:36:56.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:56.575 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:57.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:58.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:58.241 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:58.242 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:36:58.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:36:58.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:58.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:36:58.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:59.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:59.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:59.929 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:36:59.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:36:59.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:59.930 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:00.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:01.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:01.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:01.842 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:37:01.842 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:37:01.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:01.843 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:02.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:03.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:03.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:03.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:37:03.936 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:37:03.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:03.936 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:04.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:05.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:05.885 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:05.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:37:05.886 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:37:05.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:05.886 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:06.856 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:07.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:07.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:07.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:37:07.859 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:37:07.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:07.859 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:08.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:09.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:09.421 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:37:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:37:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:09.421 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:09.852 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2466782001,v1:127.0.0.1:6819/2466782001] [v2:127.0.0.1:6820/2466782001,v1:127.0.0.1:6821/2466782001] exists,up e0f0c195-3b3d-4b59-b9a4-f8ddcb87644b 2024-04-25T09:37:09.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:37:09.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:37:09.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:37:09.852 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:37:10.416 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T09:37:10.434 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:37:10.435 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:37:10.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:37:10.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:37:10.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:37:11.119 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:37:11.151 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:37:11.187 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:37:11.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:37:11.341 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:37:11.367 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:37:11.394 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:37:11.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:37:11.544 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:37:11.573 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:37:11.601 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:37:11.616 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:37:11.757 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:37:11.785 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:37:11.813 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:37:11.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:37:12.123 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:37:12.168 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:37:12.206 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:37:12.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:37:12.226 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:37:12.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:12.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:12.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:12.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:12.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:12.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:12.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:12.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:12.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:12.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:13.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:13.022 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:13.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:13.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:13.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:13.273 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:13.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:13.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:13.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:13.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:13.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:13.625 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:13.725 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:13.826 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:13.827 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:13.943 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:13.944 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:14.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:14.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:14.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:14.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:14.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:14.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:14.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:14.403 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:14.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:14.504 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:14.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:14.613 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:37:14.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:37:14.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:37:14.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:37:14.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:37:14.731 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:37:14.876 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836501 2024-04-25T09:37:14.900 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672974 2024-04-25T09:37:14.925 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509446 2024-04-25T09:37:14.939 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:37:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:17.942 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:37:18.107 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:37:18.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:37:18.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:18.108 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:37:18.267 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:37:18.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:18.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:37:18.416 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:37:18.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:18.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:37:18.416 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:37:18.569 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:37:18.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:37:18.692 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:37:18.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:18.693 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:18.813 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:18.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:18.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:18.914 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:18.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:18.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:19.039 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:19.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:19.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:19.148 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:19.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:19.151 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:19.282 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:19.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:19.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:19.383 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:19.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:19.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:19.484 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:19.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:19.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:19.584 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:19.588 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:19.710 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:19.713 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:19.714 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:19.844 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:19.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:19.847 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:19.960 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:19.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:19.964 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:20.061 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:20.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:20.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:20.170 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:20.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:20.174 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:20.287 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:20.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:20.291 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:20.388 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:20.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:20.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:20.488 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:20.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:20.493 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:20.597 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:20.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:20.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:20.723 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:20.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:20.726 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:20.832 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:20.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:37:20.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:37:20.991 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:37:20.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:37:20.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:37:20.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:37:20.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:37:20.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:37:21.140 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836504 2024-04-25T09:37:21.165 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672977 2024-04-25T09:37:21.190 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509450 2024-04-25T09:37:21.204 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:37:24.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:24.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:37:24.371 INFO:tasks.workunit.client.0.smithi102.stdout:21 2024-04-25T09:37:24.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:24.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:37:24.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:37:24.528 INFO:tasks.workunit.client.0.smithi102.stdout:21 2024-04-25T09:37:24.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:37:24.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:24.683 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:37:24.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:24.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:37:24.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:37:24.839 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:37:24.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: ceph pg 1.0 query 2024-04-25T09:37:24.840 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: jq .info.stats.up_primary 2024-04-25T09:37:24.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: local PRIMARY=1 2024-04-25T09:37:24.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:111: TEST_repro_long_log2: kill_daemons td/repro-long-log TERM osd.1 2024-04-25T09:37:24.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:37:24.995 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:37:24.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:37:24.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:37:24.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:37:25.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:37:25.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: CEPH_ARGS='--osd-max-pg-log-entries=2 --osd-pg-log-dups-tracked=3 --no-mon-config' 2024-04-25T09:37:25.303 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: ceph-objectstore-tool --data-path td/repro-long-log/1 --pgid 1.0 --op trim-pg-log 2024-04-25T09:37:26.413 INFO:tasks.workunit.client.0.smithi102.stderr:Log bounds are: (19'20,19'41] 2024-04-25T09:37:26.414 INFO:tasks.workunit.client.0.smithi102.stdout:Removing keys 0000000019.00000000000000000021 - 0000000019.00000000000000000039 2024-04-25T09:37:26.450 INFO:tasks.workunit.client.0.smithi102.stderr:Finished trimming, now compacting... 2024-04-25T09:37:27.430 INFO:tasks.workunit.client.0.smithi102.stdout:Finished trimming pg log 2024-04-25T09:37:27.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:113: TEST_repro_long_log2: activate_osd td/repro-long-log 1 2024-04-25T09:37:27.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repro-long-log 2024-04-25T09:37:27.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:37:27.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T09:37:27.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:37:27.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:37:27.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:37:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:37:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:37:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:37:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:37:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:37:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:37:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:37:27.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:37:27.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:37:27.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:37:27.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:37:27.955 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:37:27.955 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:37:27.955 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:37:27.955 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:37:27.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:37:27.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:37:27.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:37:27.955 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:37:27.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:37:27.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:37:27.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:37:27.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:37:27.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:37:27.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:37:27.956 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:37:27.956 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:37:27.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T09:37:27.957 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:37:27.957 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repro-long-log/1/whoami 2024-04-25T09:37:27.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T09:37:27.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:37:27.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:37:27.958 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:37:27.992 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:28.003+0000 7fe8ef60e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:27.995 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:28.007+0000 7fe8ef60e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:27.996 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:28.007+0000 7fe8ef60e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:28.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T09:37:28.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:37:28.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:37:28.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:37:28.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:37:28.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:28.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:37:28.383 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:37:28.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:28.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:28.558 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:28.571+0000 7fe8ef60e7c0 -1 Falling back to public interface 2024-04-25T09:37:28.816 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:29.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:29.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:29.818 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:37:29.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:37:29.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:29.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:30.226 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:30.946 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:30.959+0000 7fe8ef60e7c0 -1 osd.1 19 log_to_monitors true 2024-04-25T09:37:31.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:31.227 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:31.228 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:37:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:37:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:31.228 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:31.674 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [10,19) [v2:127.0.0.1:6810/1663993351,v1:127.0.0.1:6811/1663993351] [v2:127.0.0.1:6812/1663993351,v1:127.0.0.1:6813/1663993351] exists,up aad66ef2-8259-4b99-8047-20faf315b1a3 2024-04-25T09:37:31.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:37:31.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:37:31.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:37:31.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:114: TEST_repro_long_log2: wait_for_clean 2024-04-25T09:37:31.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:37:31.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:37:31.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:37:31.676 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:37:31.676 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:37:31.677 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:37:31.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:37:31.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:37:31.677 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:37:31.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:37:31.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:37:31.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:37:31.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:37:31.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:37:31.777 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:37:32.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:37:32.184 INFO:tasks.workunit.client.0.smithi102.stderr:1 2024-04-25T09:37:32.184 INFO:tasks.workunit.client.0.smithi102.stderr:2' 2024-04-25T09:37:32.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:37:32.184 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:37:32.184 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:37:32.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836508 2024-04-25T09:37:32.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836508 2024-04-25T09:37:32.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508' 2024-04-25T09:37:32.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:37:32.391 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:37:32.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=98784247811 2024-04-25T09:37:32.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 98784247811 2024-04-25T09:37:32.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-98784247811' 2024-04-25T09:37:32.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:37:32.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:37:32.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509454 2024-04-25T09:37:32.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509454 2024-04-25T09:37:32.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-98784247811 2-64424509454' 2024-04-25T09:37:32.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:37:32.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836508 2024-04-25T09:37:32.702 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:37:32.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:37:32.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-04-25T09:37:32.703 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:37:32.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836508 2024-04-25T09:37:32.704 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-04-25T09:37:32.704 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.0 seq 21474836508 2024-04-25T09:37:32.705 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:37:33.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836508 2024-04-25T09:37:33.092 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:37:33.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-98784247811 2024-04-25T09:37:33.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:37:33.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:37:33.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-98784247811 2024-04-25T09:37:33.094 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:37:33.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=98784247811 2024-04-25T09:37:33.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 98784247811' 2024-04-25T09:37:33.096 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.1 seq 98784247811 2024-04-25T09:37:33.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:37:33.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 98784247811 -lt 98784247811 2024-04-25T09:37:33.559 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:37:33.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509454 2024-04-25T09:37:33.561 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:37:33.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:37:33.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509454 2024-04-25T09:37:33.563 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:37:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509454 2024-04-25T09:37:33.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-04-25T09:37:33.565 INFO:tasks.workunit.client.0.smithi102.stdout:waiting osd.2 seq 64424509454 2024-04-25T09:37:33.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:37:33.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509452 -lt 64424509454 2024-04-25T09:37:33.980 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:37:34.982 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:37:34.982 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:37:35.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509454 2024-04-25T09:37:35.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:37:35.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:37:35.420 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:37:35.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:37:35.972 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:37:35.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:37:35.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:37:35.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:37:35.972 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:37:35.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:37:35.973 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:37:36.363 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:37:36.363 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:37:36.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:37:36.364 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:37:36.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:37:36.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:37:36.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:37:36.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:115: TEST_repro_long_log2: test_log_size 1.0 21 18 2024-04-25T09:37:36.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:37:36.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:37:36.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-04-25T09:37:36.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:37:37.050 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836511 2024-04-25T09:37:37.072 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 98784247814 2024-04-25T09:37:37.095 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509457 2024-04-25T09:37:37.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:37:40.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:40.111 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:37:40.276 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:37:40.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:40.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:37:40.277 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:37:40.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:40.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:37:40.592 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:37:40.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:37:40.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:37:40.593 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-04-25T09:37:40.750 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:37:40.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:37:40.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:37:40.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:37:40.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:37:40.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:37:40.752 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:37:40.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:37:40.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:37:40.752 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:37:41.061 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:37:41.062 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:37:41.063 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:37:41.063 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:37:41.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:37:41.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:37:41.066 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:37:41.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:37:41.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:37:41.068 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:37:41.068 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:37:41.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:37:41.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:37:41.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:37:41.092 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:37:41.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:37:41.093 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:37:41.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493206 2024-04-25T09:37:41.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:37:41.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:37:41.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:37:41.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:37:41.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:37:41.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:37:41.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:37:41.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:37:41.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:37:41.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:37:41.096 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:37:41.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:37:41.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:37:41.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:37:41.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:37:41.097 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:37:41.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:37:41.099 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:37:41.099 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:37:41.100 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:37:41.100 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:37:41.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:37:41.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:37:41.102 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:37:41.102 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:37:41.103 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:37:41.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:37:41.104 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:37:41.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:37:41.105 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:37:41.106 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:37:41.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493206 2024-04-25T09:37:41.106 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:37:41.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:37:41.107 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:37:41.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:37:41.108 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:37:41.109 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:37:41.109 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.493206 2024-04-25T09:37:41.110 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:37:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:37:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:37:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:37:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries td/repro-long-log 2024-04-25T09:37:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:120: TEST_trim_max_entries: local dir=td/repro-long-log 2024-04-25T09:37:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:122: TEST_trim_max_entries: setup_log_test td/repro-long-log 2024-04-25T09:37:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:37:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:37:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:37:41.112 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:37:41.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:37:41.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:37:41.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:37:41.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:37:41.113 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:37:41.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:37:41.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:37:41.566 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:37:41.567 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:37:41.567 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:37:41.567 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:37:41.567 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:37:41.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:37:41.770 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:37:41.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:37:41.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:37:41.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:37:41.771 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:37:41.772 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:37:41.772 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:37:41.772 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:37:41.773 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:37:41.773 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:37:41.773 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:37:41.773 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:37:41.773 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493206/ceph-mon.a.asok 2024-04-25T09:37:41.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:37:41.773 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493206/ceph-mon.a.asok config get fsid 2024-04-25T09:37:41.893 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:37:41.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:37:41.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:37:41.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:37:41.894 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:37:41.894 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:37:41.894 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:37:41.894 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:37:41.895 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:37:41.895 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:37:41.895 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:37:41.895 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493206/ceph-mon.a.asok 2024-04-25T09:37:41.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:37:41.895 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493206/ceph-mon.a.asok config get mon_host 2024-04-25T09:37:42.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:37:42.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:37:42.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:37:42.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:37:42.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:37:42.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:37:42.004 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:37:42.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:37:42.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:37:42.916 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:37:42.917 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:37:42.917 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:37:42.917 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:37:42.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:37:42.917 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:37:42.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:37:42.950 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:37:42.951 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:37:42.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:37:42.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:37:42.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:37:42.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:37:42.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:37:42.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:37:42.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:37:42.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:37:42.952 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:37:42.952 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:37:42.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=42fc6c76-7ac8-4d51-ad8e-b5236e7dbc4c 2024-04-25T09:37:42.953 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 42fc6c76-7ac8-4d51-ad8e-b5236e7dbc4c' 2024-04-25T09:37:42.953 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 42fc6c76-7ac8-4d51-ad8e-b5236e7dbc4c 2024-04-25T09:37:42.953 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:37:42.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBmJCpm6heBOhAAjVUpYA/XqGB9Ls2loJFYuQ== 2024-04-25T09:37:42.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBmJCpm6heBOhAAjVUpYA/XqGB9Ls2loJFYuQ=="}' 2024-04-25T09:37:42.968 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 42fc6c76-7ac8-4d51-ad8e-b5236e7dbc4c -i td/repro-long-log/0/new.json 2024-04-25T09:37:43.255 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:37:43.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:37:43.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBmJCpm6heBOhAAjVUpYA/XqGB9Ls2loJFYuQ== --osd-uuid 42fc6c76-7ac8-4d51-ad8e-b5236e7dbc4c 2024-04-25T09:37:43.298 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:43.311+0000 7fea89bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:43.315 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:43.327+0000 7fea89bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:43.316 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:43.327+0000 7fea89bad7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:43.317 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:43.327+0000 7fea89bad7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:37:43.317 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:43.327+0000 7fea89bad7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:37:43.317 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:43.327+0000 7fea89bad7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:37:43.347 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:43.359+0000 7fea89bad7c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:37:49.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:37:49.243 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:37:49.245 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:37:49.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:37:49.245 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:37:49.823 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:37:49.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:37:49.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:37:49.823 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:37:49.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:37:49.824 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:37:49.852 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:49.863+0000 7fe4325657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:49.854 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:49.867+0000 7fe4325657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:49.855 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:49.867+0000 7fe4325657c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:50.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:37:50.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:37:50.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:37:50.255 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:37:50.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:37:50.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:37:50.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:50.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:37:50.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:50.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:37:50.403 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:50.415+0000 7fe4325657c0 -1 Falling back to public interface 2024-04-25T09:37:50.678 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:51.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:51.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:51.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:37:51.680 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:37:51.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:51.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:37:52.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:52.917 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:37:52.926+0000 7fe4325657c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:37:53.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:53.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:53.090 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:37:53.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:37:53.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:53.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:37:53.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:54.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:54.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:54.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:37:54.657 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:37:54.657 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:54.658 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:37:55.094 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:56.096 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:56.096 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:37:56.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:56.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:37:56.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:56.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:37:56.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:57.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:57.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:57.518 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:37:57.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:37:57.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:57.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:37:58.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:59.013 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:37:59.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:59.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:59.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:37:59.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:59.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:37:59.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:00.455 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:00.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:00.456 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:38:00.456 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:38:00.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:00.457 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:38:01.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:02.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:02.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:02.018 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:38:02.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:38:02.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:02.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:38:02.440 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:03.441 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:03.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:03.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:38:03.442 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:38:03.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:03.442 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:38:03.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:04.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:04.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:04.989 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:38:04.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:38:04.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:04.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:38:05.632 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:06.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:06.634 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:06.634 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:38:06.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:38:06.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:06.635 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:38:07.700 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:08.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:08.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:08.703 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:38:08.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:38:08.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:08.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:38:09.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:10.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:10.452 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:10.453 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:38:10.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:38:10.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:10.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:38:11.163 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:12.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:12.165 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:12.165 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:38:12.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:38:12.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:12.166 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:38:12.763 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:13.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:13.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:13.765 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:38:13.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:38:13.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:13.766 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:38:14.699 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:15.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:15.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:15.701 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:38:15.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:38:15.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:15.702 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:38:16.230 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1836477610,v1:127.0.0.1:6803/1836477610] [v2:127.0.0.1:6804/1836477610,v1:127.0.0.1:6805/1836477610] exists,up 42fc6c76-7ac8-4d51-ad8e-b5236e7dbc4c 2024-04-25T09:38:16.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:38:16.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:38:16.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:38:16.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:38:16.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:38:16.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:38:16.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:38:16.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:38:16.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:38:16.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:38:16.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:38:16.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:38:16.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:38:16.235 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:38:16.236 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 be8aeb22-0dde-46b8-9897-37d2d566b4c4 2024-04-25T09:38:16.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be8aeb22-0dde-46b8-9897-37d2d566b4c4 2024-04-25T09:38:16.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 be8aeb22-0dde-46b8-9897-37d2d566b4c4' 2024-04-25T09:38:16.236 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:38:16.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIJCpmfOSsDxAAh5eOa02+SUB7qqrlHjuhvg== 2024-04-25T09:38:16.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIJCpmfOSsDxAAh5eOa02+SUB7qqrlHjuhvg=="}' 2024-04-25T09:38:16.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be8aeb22-0dde-46b8-9897-37d2d566b4c4 -i td/repro-long-log/1/new.json 2024-04-25T09:38:16.761 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:38:16.777 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:38:16.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIJCpmfOSsDxAAh5eOa02+SUB7qqrlHjuhvg== --osd-uuid be8aeb22-0dde-46b8-9897-37d2d566b4c4 2024-04-25T09:38:16.803 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:16.814+0000 7f4e3efc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:16.805 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:16.814+0000 7f4e3efc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:16.806 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:16.818+0000 7f4e3efc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:16.806 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:16.818+0000 7f4e3efc67c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:38:16.807 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:16.818+0000 7f4e3efc67c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:38:16.807 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:16.818+0000 7f4e3efc67c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:38:16.883 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:16.894+0000 7f4e3efc67c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:38:23.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:38:23.007 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:38:23.008 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:38:23.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:38:23.008 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:38:23.563 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:38:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:38:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:38:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:38:23.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:38:23.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:38:23.593 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:23.602+0000 7fdbfba0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:23.606 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:23.614+0000 7fdbfba0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:23.607 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:23.618+0000 7fdbfba0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:23.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:38:23.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:38:23.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:38:23.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:38:23.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:38:23.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:23.971 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:38:23.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:38:23.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:23.971 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:24.179 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:24.190+0000 7fdbfba0b7c0 -1 Falling back to public interface 2024-04-25T09:38:24.410 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:25.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:25.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:25.412 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:38:25.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:38:25.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:25.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:25.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:26.615 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:26.626+0000 7fdbfba0b7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:38:26.817 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:26.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:26.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:38:26.818 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:38:26.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:26.818 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:27.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:28.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:38:28.389 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:38:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:28.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:28.830 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:29.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:29.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:29.832 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:38:29.833 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:38:29.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:29.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:30.398 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:31.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:31.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:31.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:38:31.400 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:38:31.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:31.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:31.855 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:32.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:32.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:32.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:38:32.857 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:38:32.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:32.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:33.369 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:34.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:34.371 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:34.372 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:38:34.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:38:34.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:34.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:34.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:35.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:35.803 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:35.804 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:38:35.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:38:35.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:35.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:36.295 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:37.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:37.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:37.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:38:37.298 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:38:37.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:37.298 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:37.715 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:38.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:38.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:38.717 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:38:38.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:38:38.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:38.717 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:39.123 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:40.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:40.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:40.125 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:38:40.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:38:40.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:40.126 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:40.924 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:41.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:41.925 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:41.926 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:38:41.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:38:41.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:41.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:42.665 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:43.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:43.670 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:43.670 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:38:43.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:38:43.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:43.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:44.451 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:45.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:45.453 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:45.454 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:38:45.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:38:45.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:45.454 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:45.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:46.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:46.874 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:46.875 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:38:46.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:38:46.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:46.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:47.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:48.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:48.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:48.640 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:38:48.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:38:48.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:48.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:49.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:50.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:50.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:50.307 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:38:50.308 INFO:tasks.workunit.client.0.smithi102.stdout:17 2024-04-25T09:38:50.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:50.308 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:38:50.807 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3412331906,v1:127.0.0.1:6811/3412331906] [v2:127.0.0.1:6812/3412331906,v1:127.0.0.1:6813/3412331906] exists,up be8aeb22-0dde-46b8-9897-37d2d566b4c4 2024-04-25T09:38:50.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:38:50.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:38:50.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:38:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:38:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:38:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:38:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:38:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:38:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:38:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:38:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:38:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:38:50.813 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:38:50.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:38:50.814 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:38:50.814 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5caf3c10-eb81-44d6-9ba5-6691a12d3c4e 2024-04-25T09:38:50.814 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 5caf3c10-eb81-44d6-9ba5-6691a12d3c4e 2024-04-25T09:38:50.815 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5caf3c10-eb81-44d6-9ba5-6691a12d3c4e' 2024-04-25T09:38:50.815 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:38:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCqJCpms0Y6MhAAPMuKtV1ZzbmM1QgxgAQqSw== 2024-04-25T09:38:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCqJCpms0Y6MhAAPMuKtV1ZzbmM1QgxgAQqSw=="}' 2024-04-25T09:38:50.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5caf3c10-eb81-44d6-9ba5-6691a12d3c4e -i td/repro-long-log/2/new.json 2024-04-25T09:38:51.315 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:38:51.331 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:38:51.332 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqJCpms0Y6MhAAPMuKtV1ZzbmM1QgxgAQqSw== --osd-uuid 5caf3c10-eb81-44d6-9ba5-6691a12d3c4e 2024-04-25T09:38:51.364 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:51.373+0000 7f3de17727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:51.369 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:51.377+0000 7f3de17727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:51.370 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:51.377+0000 7f3de17727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:51.370 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:51.377+0000 7f3de17727c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:38:51.371 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:51.377+0000 7f3de17727c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:38:51.371 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:51.377+0000 7f3de17727c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:38:51.408 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:51.417+0000 7f3de17727c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:38:57.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:38:57.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:38:57.489 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T09:38:57.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:38:57.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:38:58.033 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:38:58.033 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:38:58.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:38:58.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:38:58.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:38:58.034 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:38:58.062 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:58.069+0000 7f4db66fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:58.064 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:58.073+0000 7f4db66fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:58.065 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:58.073+0000 7f4db66fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:58.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:38:58.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:38:58.473 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:38:58.473 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:38:58.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:38:58.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:38:58.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:58.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:38:58.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:58.474 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:38:58.609 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:38:58.617+0000 7f4db66fe7c0 -1 Falling back to public interface 2024-04-25T09:38:58.902 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:59.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:59.904 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:59.904 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:38:59.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:38:59.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:59.905 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:00.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:01.088 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:39:01.097+0000 7f4db66fe7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:39:01.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:01.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:01.381 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:39:01.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:39:01.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:01.382 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:01.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:02.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:02.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:02.894 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:39:02.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:39:02.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:02.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:03.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:04.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:04.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:04.318 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:39:04.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:39:04.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:04.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:04.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:05.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:05.893 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:05.893 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:39:05.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:39:05.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:05.894 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:06.370 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:07.372 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:07.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:07.373 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:39:07.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:39:07.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:07.373 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:07.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:08.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:08.934 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:08.934 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:39:08.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:39:08.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:08.935 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:09.352 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:10.354 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:10.354 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:39:10.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:39:10.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:10.355 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:10.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:11.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:11.919 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:11.919 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:39:11.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:39:11.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:11.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:12.381 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:13.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:13.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:13.384 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:39:13.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:39:13.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:13.385 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:14.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:15.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:15.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:15.048 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:39:15.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:39:15.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:15.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:15.783 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:16.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:16.784 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:16.785 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:39:16.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:39:16.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:16.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:17.494 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:18.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:18.496 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:18.496 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:39:18.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:39:18.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:18.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:19.168 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:20.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:20.169 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:20.170 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:39:20.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:39:20.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:20.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:21.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:22.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:22.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:22.236 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:39:22.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:39:22.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:22.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:22.938 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:23.940 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:23.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:23.941 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:39:23.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:39:23.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:23.941 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:39:24.435 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2173995429,v1:127.0.0.1:6819/2173995429] [v2:127.0.0.1:6820/2173995429,v1:127.0.0.1:6821/2173995429] exists,up 5caf3c10-eb81-44d6-9ba5-6691a12d3c4e 2024-04-25T09:39:24.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:39:24.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:39:24.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:39:24.436 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:39:24.931 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T09:39:24.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:39:24.948 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:39:25.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:39:25.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:39:25.358 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:39:25.491 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:39:25.517 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:39:25.543 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:39:25.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:39:25.694 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:39:25.719 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:39:25.745 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:39:25.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:39:25.942 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:39:25.972 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:39:26.002 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:39:26.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:39:26.165 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:39:26.191 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:39:26.218 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:39:26.232 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:39:26.450 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:39:26.482 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:39:26.516 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:39:26.532 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:39:26.534 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:39:26.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:26.534 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:26.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:26.760 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:27.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:27.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:27.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:27.153 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:27.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:27.286 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:27.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:27.412 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:27.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:27.530 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:27.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:27.664 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:27.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:27.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:27.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:27.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:27.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:27.990 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:28.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:28.142 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:28.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:28.250 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:28.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:28.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:28.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:28.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:28.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:28.627 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:28.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:28.727 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:28.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:28.828 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:28.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:28.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:29.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:29.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:39:29.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:39:29.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:39:29.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:39:29.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:39:29.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:39:29.276 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836500 2024-04-25T09:39:29.300 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672972 2024-04-25T09:39:29.324 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509446 2024-04-25T09:39:29.339 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:39:32.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:32.342 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:32.506 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:39:32.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:32.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:32.507 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:39:32.659 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:39:32.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:32.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:32.820 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:32.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:32.820 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:32.821 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:39:32.973 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:32.973 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:39:33.142 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:39:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:33.143 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:33.263 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:33.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:33.266 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:33.371 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:33.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:33.375 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:33.480 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:33.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:33.483 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:33.597 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:33.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:33.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:33.698 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:33.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:33.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:33.798 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:33.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:33.802 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:33.924 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:33.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:33.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:34.025 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:34.028 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:34.029 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:34.159 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:34.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:34.162 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:34.293 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:34.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:34.297 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:34.402 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:34.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:34.405 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:34.502 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:34.505 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:34.506 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:34.637 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:34.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:34.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:34.745 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:34.748 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:34.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:34.854 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:34.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:34.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:34.988 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:34.991 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:34.992 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:35.089 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:35.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:35.093 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:35.189 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:35.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:35.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:35.289 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:35.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:39:35.293 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:39:35.423 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:35.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:39:35.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:39:35.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:39:35.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:39:35.429 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:39:35.575 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836503 2024-04-25T09:39:35.599 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672976 2024-04-25T09:39:35.623 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509450 2024-04-25T09:39:35.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:39:38.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:38.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:38.806 INFO:tasks.workunit.client.0.smithi102.stdout:21 2024-04-25T09:39:38.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:38.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:38.807 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:39:38.969 INFO:tasks.workunit.client.0.smithi102.stdout:21 2024-04-25T09:39:38.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:38.970 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:39.127 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:39.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:39.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:39.128 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:39:39.278 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:39.278 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:124: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-04-25T09:39:39.418 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:39:39.447 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:39:39.475 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:39:39.490 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:125: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-04-25T09:39:39.633 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:39:39.661 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:39:39.691 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:39:39.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:126: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-04-25T09:39:39.851 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:39:39.879 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:39:39.906 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:39:39.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:127: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-04-25T09:39:40.105 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:39:40.133 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:39:40.166 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:39:40.181 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:128: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 0 2024-04-25T09:39:40.324 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:39:40.353 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:39:40.385 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:39:40.399 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:131: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:39:40.539 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:132: TEST_trim_max_entries: test_log_size 1.0 18 2024-04-25T09:39:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:39:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-04-25T09:39:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:39:40.543 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:39:40.688 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836506 2024-04-25T09:39:40.712 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672979 2024-04-25T09:39:40.736 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509453 2024-04-25T09:39:40.751 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:39:43.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:43.753 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:43.919 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:39:43.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:43.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:43.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-04-25T09:39:44.117 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:39:44.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:44.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:44.271 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:44.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:44.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:44.272 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:39:44.425 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:44.426 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:133: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:39:44.533 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:44.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:134: TEST_trim_max_entries: test_log_size 1.0 15 2024-04-25T09:39:44.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:39:44.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-04-25T09:39:44.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:39:44.537 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:39:44.680 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836509 2024-04-25T09:39:44.705 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672981 2024-04-25T09:39:44.728 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509456 2024-04-25T09:39:44.743 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:39:47.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:47.746 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:47.912 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:39:47.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:47.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:47.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-04-25T09:39:48.066 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:39:48.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:48.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:48.220 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:48.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:48.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:48.221 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:39:48.378 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:48.379 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:135: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:39:48.536 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:48.539 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:136: TEST_trim_max_entries: test_log_size 1.0 12 2024-04-25T09:39:48.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:39:48.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-04-25T09:39:48.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:39:48.540 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:39:48.683 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836512 2024-04-25T09:39:48.707 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672984 2024-04-25T09:39:48.732 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509458 2024-04-25T09:39:48.747 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:39:51.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:51.750 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:51.915 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:39:51.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:51.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:51.916 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-04-25T09:39:52.104 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:39:52.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:52.105 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:52.258 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:52.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:52.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:52.259 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:39:52.410 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:52.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:137: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:39:52.530 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:52.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:138: TEST_trim_max_entries: test_log_size 1.0 9 2024-04-25T09:39:52.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:39:52.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-04-25T09:39:52.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:39:52.535 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:39:52.671 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836515 2024-04-25T09:39:52.695 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672987 2024-04-25T09:39:52.720 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509461 2024-04-25T09:39:52.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:39:55.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:55.739 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:55.899 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:39:55.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:55.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:55.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-04-25T09:39:56.052 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:39:56.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:56.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:56.193 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:56.193 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:39:56.194 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:39:56.350 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:39:56.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:139: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:39:56.475 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:39:56.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:140: TEST_trim_max_entries: test_log_size 1.0 6 2024-04-25T09:39:56.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:39:56.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-04-25T09:39:56.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:39:56.478 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:39:56.618 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836518 2024-04-25T09:39:56.640 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672990 2024-04-25T09:39:56.664 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509464 2024-04-25T09:39:56.677 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:39:59.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:59.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:59.844 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:39:59.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:39:59.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:39:59.845 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-04-25T09:40:00.003 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:40:00.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:00.003 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:00.169 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:00.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:00.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:00.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:40:00.322 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:00.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:141: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:40:00.461 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:40:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:142: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:40:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:40:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:40:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:40:00.465 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:40:00.609 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836520 2024-04-25T09:40:00.632 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672993 2024-04-25T09:40:00.657 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509467 2024-04-25T09:40:00.671 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:40:03.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:03.674 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:40:03.839 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:40:03.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:03.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:40:03.840 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:40:03.995 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:40:03.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:03.995 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:04.171 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:04.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:04.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:04.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:40:04.327 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:04.327 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:145: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:40:04.431 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:40:04.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:146: TEST_trim_max_entries: test_log_size 1.0 4 2024-04-25T09:40:04.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:40:04.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:40:04.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:40:04.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:40:04.576 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836523 2024-04-25T09:40:04.602 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672995 2024-04-25T09:40:04.626 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509470 2024-04-25T09:40:04.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:40:07.643 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:07.644 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:40:07.811 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:40:07.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:07.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:40:07.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:40:07.967 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:40:07.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:07.967 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:08.157 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:08.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:08.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:08.157 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:40:08.311 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:08.311 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:147: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:40:08.424 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:40:08.427 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:148: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:40:08.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:40:08.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:40:08.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:40:08.428 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:40:08.567 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836526 2024-04-25T09:40:08.592 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672998 2024-04-25T09:40:08.616 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509472 2024-04-25T09:40:08.631 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:40:11.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:11.633 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:40:11.798 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:40:11.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:11.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:40:11.799 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:40:11.960 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:40:11.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:11.961 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:12.191 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:12.192 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:40:12.351 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:12.351 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:149: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:40:12.444 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:40:12.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:150: TEST_trim_max_entries: test_log_size 1.0 4 2024-04-25T09:40:12.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:40:12.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:40:12.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:40:12.448 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:40:12.589 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836529 2024-04-25T09:40:12.613 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949673001 2024-04-25T09:40:12.638 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509475 2024-04-25T09:40:12.653 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:40:15.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:15.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:40:15.822 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:40:15.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:15.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:40:15.822 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:40:15.975 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:40:15.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:15.976 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:16.129 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:16.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:16.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:16.130 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:40:16.279 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:16.280 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:151: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:40:16.397 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:40:16.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:152: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:40:16.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:40:16.400 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:40:16.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:40:16.401 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:40:16.536 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836532 2024-04-25T09:40:16.559 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949673004 2024-04-25T09:40:16.583 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509478 2024-04-25T09:40:16.598 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:40:19.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:19.601 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:40:19.766 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:40:19.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:19.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:40:19.767 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:40:19.919 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:40:19.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:19.920 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:20.066 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:20.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:40:20.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:40:20.067 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:40:20.212 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:20.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:40:20.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:40:20.213 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:40:20.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:40:20.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:40:20.214 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:40:20.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:40:20.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:40:20.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:40:20.523 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:40:20.523 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:40:20.524 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:40:20.525 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:40:20.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:40:20.526 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:40:20.526 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:40:20.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:40:20.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:40:20.528 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:40:20.528 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:40:20.529 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:40:20.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:40:20.531 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:40:20.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:40:20.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:40:20.553 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:40:20.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493206 2024-04-25T09:40:20.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:40:20.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:40:20.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:40:20.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:40:20.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:40:20.555 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:40:20.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:40:20.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:40:20.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:40:20.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:40:20.556 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:40:20.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:40:20.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:40:20.556 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:40:20.557 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:40:20.557 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:40:20.558 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:40:20.559 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:40:20.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:40:20.560 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:40:20.560 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:40:20.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:40:20.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:40:20.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:40:20.562 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:40:20.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:40:20.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:40:20.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:40:20.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:40:20.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:40:20.565 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:40:20.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493206 2024-04-25T09:40:20.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:40:20.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:40:20.566 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:40:20.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:40:20.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:40:20.568 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:40:20.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.493206 2024-04-25T09:40:20.571 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:40:20.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:40:20.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:40:20.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:40:20.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries_with_dups td/repro-long-log 2024-04-25T09:40:20.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:157: TEST_trim_max_entries_with_dups: local dir=td/repro-long-log 2024-04-25T09:40:20.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:159: TEST_trim_max_entries_with_dups: setup_log_test td/repro-long-log 2024-04-25T09:40:20.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:40:20.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:40:20.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:40:20.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:40:20.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:40:20.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:40:20.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:40:20.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:40:20.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:40:21.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:40:21.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:40:21.051 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:40:21.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:40:21.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:40:21.052 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:40:21.052 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:40:21.052 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:40:21.254 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:40:21.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:40:21.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:40:21.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:40:21.255 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:40:21.256 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:40:21.256 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:40:21.256 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:40:21.256 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:40:21.257 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:40:21.257 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:40:21.257 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:40:21.257 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493206/ceph-mon.a.asok 2024-04-25T09:40:21.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:40:21.258 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493206/ceph-mon.a.asok config get fsid 2024-04-25T09:40:21.374 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.493206/ceph-mon.a.asok 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:40:21.375 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.493206/ceph-mon.a.asok config get mon_host 2024-04-25T09:40:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:40:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:40:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:40:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:40:21.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:40:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:40:21.489 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:40:22.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:40:22.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:40:22.382 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:40:22.383 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:40:22.383 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:40:22.383 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:40:22.383 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:40:22.384 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:40:22.386 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:40:22.417 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:40:22.418 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:40:22.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:40:22.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:40:22.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:40:22.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:40:22.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:40:22.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:40:22.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:40:22.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:40:22.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:40:22.419 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:40:22.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f1e3e7e5-a398-4164-ba4d-b999b1d82f32 2024-04-25T09:40:22.420 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f1e3e7e5-a398-4164-ba4d-b999b1d82f32' 2024-04-25T09:40:22.421 INFO:tasks.workunit.client.0.smithi102.stdout:add osd0 f1e3e7e5-a398-4164-ba4d-b999b1d82f32 2024-04-25T09:40:22.421 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:40:22.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAGJSpmgMZ1GhAABhe7Iifucl4ITG2H6F9SDg== 2024-04-25T09:40:22.434 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAGJSpmgMZ1GhAABhe7Iifucl4ITG2H6F9SDg=="}' 2024-04-25T09:40:22.435 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f1e3e7e5-a398-4164-ba4d-b999b1d82f32 -i td/repro-long-log/0/new.json 2024-04-25T09:40:22.724 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:22.740 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:40:22.741 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGJSpmgMZ1GhAABhe7Iifucl4ITG2H6F9SDg== --osd-uuid f1e3e7e5-a398-4164-ba4d-b999b1d82f32 2024-04-25T09:40:22.768 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:22.775+0000 7f71eeeab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:22.793 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:22.803+0000 7f71eeeab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:22.794 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:22.803+0000 7f71eeeab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:22.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:22.803+0000 7f71eeeab7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:40:22.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:22.803+0000 7f71eeeab7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:40:22.795 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:22.803+0000 7f71eeeab7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:40:22.833 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:22.839+0000 7f71eeeab7c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:40:28.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:40:28.703 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:40:28.705 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd0 key to auth repository 2024-04-25T09:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:40:28.705 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:40:29.238 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.0 2024-04-25T09:40:29.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:40:29.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:40:29.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:40:29.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:40:29.239 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:40:29.268 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:29.275+0000 7f0f93f987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:29.281 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:29.287+0000 7f0f93f987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:29.282 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:29.291+0000 7f0f93f987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:29.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:40:29.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:40:29.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:40:29.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:40:29.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:40:29.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:29.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:40:29.655 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:40:29.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:29.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:29.839 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:29.847+0000 7f0f93f987c0 -1 Falling back to public interface 2024-04-25T09:40:30.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:31.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:31.087 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:40:31.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:40:31.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:31.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:31.485 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:32.314 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:32.323+0000 7f0f93f987c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:40:32.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:32.487 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:32.488 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:40:32.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:40:32.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:32.488 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:33.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:34.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:34.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:34.045 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:40:34.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:40:34.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:34.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:34.459 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:35.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:35.461 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:35.461 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:40:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:40:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:35.462 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:35.873 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:36.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:36.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:36.875 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:40:36.875 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:40:36.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:36.876 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:37.319 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:38.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:38.322 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:38.322 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:40:38.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:40:38.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:38.323 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:38.734 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:39.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:39.735 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:39.736 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:40:39.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:40:39.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:39.736 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:40.302 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:41.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:41.304 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:41.304 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:40:41.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:40:41.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:41.305 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:41.778 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:42.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:42.780 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:42.780 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:40:42.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:40:42.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:42.781 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:43.318 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:44.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:44.320 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:44.320 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:40:44.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:40:44.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:44.321 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:45.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:46.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:46.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:46.081 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:40:46.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:40:46.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:46.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:47.115 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:48.117 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:48.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:48.118 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:40:48.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:40:48.118 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:48.119 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:49.011 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:50.012 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:50.013 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:50.013 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:40:50.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:40:50.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:50.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:50.720 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:51.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:51.722 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:51.722 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:40:51.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:40:51.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:51.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:52.561 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:53.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:53.563 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:53.563 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:40:53.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:40:53.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:53.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:54.196 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:55.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:55.198 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:55.198 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:40:55.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:40:55.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:55.199 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:40:55.656 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/179591782,v1:127.0.0.1:6803/179591782] [v2:127.0.0.1:6804/179591782,v1:127.0.0.1:6805/179591782] exists,up f1e3e7e5-a398-4164-ba4d-b999b1d82f32 2024-04-25T09:40:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:40:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:40:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:40:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:40:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:40:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:40:55.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:40:55.661 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:40:55.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:40:55.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:40:55.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:40:55.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:40:55.663 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:40:55.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=20bef852-eabd-44d8-80e6-8833e873f768 2024-04-25T09:40:55.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 20bef852-eabd-44d8-80e6-8833e873f768' 2024-04-25T09:40:55.664 INFO:tasks.workunit.client.0.smithi102.stdout:add osd1 20bef852-eabd-44d8-80e6-8833e873f768 2024-04-25T09:40:55.664 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:40:55.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAnJSpmfVcqKRAAqa8cHNVt0NeZJNDGN2j0SA== 2024-04-25T09:40:55.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAnJSpmfVcqKRAAqa8cHNVt0NeZJNDGN2j0SA=="}' 2024-04-25T09:40:55.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 20bef852-eabd-44d8-80e6-8833e873f768 -i td/repro-long-log/1/new.json 2024-04-25T09:40:56.233 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:40:56.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:40:56.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAnJSpmfVcqKRAAqa8cHNVt0NeZJNDGN2j0SA== --osd-uuid 20bef852-eabd-44d8-80e6-8833e873f768 2024-04-25T09:40:56.283 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:56.291+0000 7ff561ce97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:56.286 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:56.295+0000 7ff561ce97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:56.287 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:56.295+0000 7ff561ce97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:56.288 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:56.295+0000 7ff561ce97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:40:56.288 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:56.295+0000 7ff561ce97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:40:56.288 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:56.295+0000 7ff561ce97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:40:56.389 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:40:56.395+0000 7ff561ce97c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:41:02.274 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:41:02.275 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:41:02.276 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd1 key to auth repository 2024-04-25T09:41:02.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:41:02.276 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:41:02.809 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.1 2024-04-25T09:41:02.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:41:02.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:41:02.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:41:02.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:41:02.810 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:41:02.838 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:02.847+0000 7f5ab54887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:02.841 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:02.847+0000 7f5ab54887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:02.842 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:02.851+0000 7f5ab54887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:03.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:41:03.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:41:03.281 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:41:03.282 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:41:03.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:41:03.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:41:03.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:03.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:41:03.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:03.283 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:03.403 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:03.411+0000 7f5ab54887c0 -1 Falling back to public interface 2024-04-25T09:41:03.716 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:04.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:04.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:04.718 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:41:04.718 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:41:04.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:04.719 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:05.173 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:06.003 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:06.010+0000 7f5ab54887c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:41:06.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:06.175 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:06.175 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:41:06.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:41:06.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:06.176 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:06.652 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:07.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:07.654 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:07.654 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:41:07.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:41:07.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:07.655 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:08.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:09.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:09.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:09.089 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:41:09.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:41:09.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:09.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:09.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:10.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:10.501 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:10.502 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:41:10.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:41:10.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:10.502 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:10.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:11.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:11.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:11.917 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:41:11.917 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:41:11.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:11.918 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:12.347 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:13.348 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:13.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:13.349 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:41:13.349 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:41:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:13.350 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:13.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:14.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:14.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:14.835 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:41:14.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:41:14.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:14.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:15.267 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:16.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:16.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:16.270 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:41:16.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:41:16.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:16.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:16.762 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:17.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:17.764 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:17.765 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:41:17.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:41:17.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:17.765 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:18.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:19.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:19.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:19.641 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:41:19.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:41:19.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:19.642 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:20.833 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:21.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:21.835 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:21.835 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:41:21.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:41:21.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:21.836 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:22.785 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:23.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:23.787 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:23.787 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:41:23.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:41:23.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:23.788 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:24.858 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:25.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:25.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:25.861 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:41:25.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:41:25.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:25.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:26.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:27.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:27.552 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:27.553 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:41:27.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:41:27.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:27.553 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:28.144 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:29.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:29.146 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:29.146 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:41:29.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:41:29.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:29.147 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:29.633 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1665337654,v1:127.0.0.1:6811/1665337654] [v2:127.0.0.1:6812/1665337654,v1:127.0.0.1:6813/1665337654] exists,up 20bef852-eabd-44d8-80e6-8833e873f768 2024-04-25T09:41:29.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:41:29.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:41:29.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:41:29.636 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:41:29.637 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:41:29.638 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:41:29.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:41:29.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:41:29.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:41:29.638 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:41:29.638 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:41:29.638 INFO:tasks.workunit.client.0.smithi102.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:41:29.638 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:41:29.639 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4062dea3-63bb-42af-936e-7438ebd5d2f2 2024-04-25T09:41:29.640 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 4062dea3-63bb-42af-936e-7438ebd5d2f2' 2024-04-25T09:41:29.641 INFO:tasks.workunit.client.0.smithi102.stdout:add osd2 4062dea3-63bb-42af-936e-7438ebd5d2f2 2024-04-25T09:41:29.641 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:41:29.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBJJSpmERPaJxAAQNwoHdenOxunmqveJpBPUw== 2024-04-25T09:41:29.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBJJSpmERPaJxAAQNwoHdenOxunmqveJpBPUw=="}' 2024-04-25T09:41:29.660 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4062dea3-63bb-42af-936e-7438ebd5d2f2 -i td/repro-long-log/2/new.json 2024-04-25T09:41:30.132 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:41:30.149 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:41:30.150 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJJSpmERPaJxAAQNwoHdenOxunmqveJpBPUw== --osd-uuid 4062dea3-63bb-42af-936e-7438ebd5d2f2 2024-04-25T09:41:30.177 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:30.182+0000 7faf2cff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:30.179 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:30.186+0000 7faf2cff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:30.180 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:30.186+0000 7faf2cff17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:30.180 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:30.186+0000 7faf2cff17c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:41:30.180 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:30.186+0000 7faf2cff17c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:41:30.180 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:30.186+0000 7faf2cff17c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:41:30.262 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:30.266+0000 7faf2cff17c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:41:36.207 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:41:36.208 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:41:36.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:41:36.209 INFO:tasks.workunit.client.0.smithi102.stdout:adding osd2 key to auth repository 2024-04-25T09:41:36.209 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:41:36.753 INFO:tasks.workunit.client.0.smithi102.stdout:start osd.2 2024-04-25T09:41:36.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:41:36.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:41:36.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=58da2f34-c09e-40a3-bbcc-3a95a0148474 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.493206/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:41:36.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:41:36.754 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:41:36.785 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:36.790+0000 7f031bf6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:36.802 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:36.806+0000 7f031bf6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:36.803 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:36.810+0000 7f031bf6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:37.170 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:41:37.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:41:37.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:41:37.171 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:41:37.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:41:37.171 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:41:37.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:37.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:41:37.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:37.172 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:37.364 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:37.370+0000 7f031bf6e7c0 -1 Falling back to public interface 2024-04-25T09:41:37.628 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:38.629 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:38.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:38.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:41:38.630 INFO:tasks.workunit.client.0.smithi102.stdout:1 2024-04-25T09:41:38.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:38.630 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:39.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:39.791 INFO:tasks.workunit.client.0.smithi102.stderr:2024-04-25T09:41:39.798+0000 7f031bf6e7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:41:40.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:40.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:40.046 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:41:40.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:41:40.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:40.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:40.568 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:41.570 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:41.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:41.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:41:41.571 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:41:41.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:41.572 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:42.095 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:43.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:43.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:43.097 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:41:43.097 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:41:43.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:43.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:43.600 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:44.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:44.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:44.602 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:41:44.602 INFO:tasks.workunit.client.0.smithi102.stdout:5 2024-04-25T09:41:44.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:44.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:45.018 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:46.019 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:46.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:46.020 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:41:46.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:41:46.020 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:46.021 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:46.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:47.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:47.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:47.564 INFO:tasks.workunit.client.0.smithi102.stdout:7 2024-04-25T09:41:47.564 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:41:47.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:47.565 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:47.986 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:48.987 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:48.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:48.988 INFO:tasks.workunit.client.0.smithi102.stdout:8 2024-04-25T09:41:48.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:41:48.988 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:48.989 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:49.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:50.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:50.413 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:50.413 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:41:50.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:41:50.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:50.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:50.862 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:51.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:51.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:51.864 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:41:51.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:41:51.864 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:51.865 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:52.518 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:53.519 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:53.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:53.520 INFO:tasks.workunit.client.0.smithi102.stdout:11 2024-04-25T09:41:53.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:41:53.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:53.520 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:54.268 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:55.269 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:55.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:55.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:41:55.270 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:41:55.270 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:55.271 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:56.053 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:57.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:57.055 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:57.056 INFO:tasks.workunit.client.0.smithi102.stdout:13 2024-04-25T09:41:57.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:41:57.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:57.057 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:57.809 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:58.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:58.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:58.811 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:41:58.812 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:41:58.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:58.812 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:41:59.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:00.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:00.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:00.682 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:42:00.683 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:42:00.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:00.683 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:01.497 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:02.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:02.499 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:02.500 INFO:tasks.workunit.client.0.smithi102.stdout:16 2024-04-25T09:42:02.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:42:02.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:02.500 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:02.993 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1398798269,v1:127.0.0.1:6819/1398798269] [v2:127.0.0.1:6820/1398798269,v1:127.0.0.1:6821/1398798269] exists,up 4062dea3-63bb-42af-936e-7438ebd5d2f2 2024-04-25T09:42:02.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:42:02.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:42:02.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:42:02.994 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:42:03.562 INFO:tasks.workunit.client.0.smithi102.stderr:pool 'test' already exists 2024-04-25T09:42:03.579 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:42:03.580 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:42:04.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:42:04.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:42:04.009 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:42:04.163 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:42:04.192 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:42:04.224 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:42:04.240 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:42:04.394 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:42:04.424 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:42:04.451 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:42:04.464 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:42:04.604 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:42:04.633 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:42:04.665 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:42:04.679 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:42:04.840 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:42:04.868 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:42:04.898 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:42:04.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:42:05.051 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:42:05.079 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:42:05.111 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:42:05.125 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:42:05.126 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:42:05.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:05.127 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:05.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:05.423 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:05.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:05.759 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:05.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:05.867 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:05.984 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:05.985 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:06.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:06.101 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:06.235 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:06.236 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:06.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:06.353 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:06.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:06.470 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:06.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:06.620 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:06.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:06.772 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:06.888 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:06.889 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:07.014 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:07.015 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:07.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:07.131 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:07.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:07.265 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:07.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:07.383 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:07.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:07.516 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:07.650 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:07.651 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:07.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:07.801 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:07.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:07.926 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:42:08.043 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:42:08.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:08.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:42:08.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:42:08.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:08.189 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836501 2024-04-25T09:42:08.213 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672972 2024-04-25T09:42:08.237 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509446 2024-04-25T09:42:08.252 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:11.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:11.418 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:11.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:11.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:11.419 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:42:11.578 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:11.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:11.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:11.737 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:42:11.737 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:11.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:11.738 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:42:11.890 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:42:11.891 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:42:12.005 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:42:12.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:12.006 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:12.118 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:12.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:12.122 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:12.252 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:12.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:12.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:12.361 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:12.364 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:12.512 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:12.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:12.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:12.637 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:12.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:12.641 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:12.746 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:12.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:12.749 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:12.896 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:12.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:12.899 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:13.013 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:13.016 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:13.017 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:13.130 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:13.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:13.134 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:13.281 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:13.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:13.285 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:13.407 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:13.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:13.411 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:13.524 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:13.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:13.527 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:13.675 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:13.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:13.680 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:13.800 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:13.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:13.804 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:13.918 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:13.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:13.921 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:14.043 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:14.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:14.047 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:14.152 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:14.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:14.156 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:14.278 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:14.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:14.282 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:14.412 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:14.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:42:14.415 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:42:14.545 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:14.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:42:14.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:14.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:42:14.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:42:14.549 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:14.694 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836504 2024-04-25T09:42:14.718 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672976 2024-04-25T09:42:14.743 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509450 2024-04-25T09:42:14.758 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:17.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:17.761 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:17.926 INFO:tasks.workunit.client.0.smithi102.stdout:21 2024-04-25T09:42:17.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:17.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:17.927 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:42:18.080 INFO:tasks.workunit.client.0.smithi102.stdout:21 2024-04-25T09:42:18.080 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:18.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:18.236 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:42:18.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:18.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:18.237 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:42:18.389 INFO:tasks.workunit.client.0.smithi102.stdout:0 2024-04-25T09:42:18.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:161: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-04-25T09:42:18.533 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:42:18.563 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:42:18.597 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:42:18.611 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:162: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-04-25T09:42:18.759 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:42:18.787 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:42:18.816 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:42:18.831 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:163: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-04-25T09:42:18.972 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:42:19.000 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:42:19.030 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:42:19.045 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:164: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-04-25T09:42:19.188 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:42:19.217 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:42:19.245 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:42:19.260 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:165: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:42:19.401 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: {} 2024-04-25T09:42:19.429 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: {} 2024-04-25T09:42:19.457 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: {} 2024-04-25T09:42:19.472 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:169: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:42:19.644 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:19.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:170: TEST_trim_max_entries_with_dups: test_log_size 1.0 18 2 2024-04-25T09:42:19.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:19.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-04-25T09:42:19.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=2 2024-04-25T09:42:19.648 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:19.794 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836508 2024-04-25T09:42:19.818 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672979 2024-04-25T09:42:19.842 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509453 2024-04-25T09:42:19.857 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:22.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:22.860 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:23.082 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:42:23.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:23.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:23.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-04-25T09:42:23.241 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:42:23.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:23.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:23.394 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:42:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:23.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 2 2024-04-25T09:42:23.570 INFO:tasks.workunit.client.0.smithi102.stdout:2 2024-04-25T09:42:23.571 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:171: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:42:23.697 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:172: TEST_trim_max_entries_with_dups: test_log_size 1.0 15 6 2024-04-25T09:42:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-04-25T09:42:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=6 2024-04-25T09:42:23.701 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:23.846 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836510 2024-04-25T09:42:23.871 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672982 2024-04-25T09:42:23.895 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509456 2024-04-25T09:42:23.910 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:26.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:26.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:27.083 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:42:27.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:27.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:27.083 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-04-25T09:42:27.241 INFO:tasks.workunit.client.0.smithi102.stdout:15 2024-04-25T09:42:27.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:27.242 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:27.393 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:42:27.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:27.393 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:27.394 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 6 2024-04-25T09:42:27.580 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:42:27.580 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:173: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:42:27.708 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:27.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:174: TEST_trim_max_entries_with_dups: test_log_size 1.0 12 10 2024-04-25T09:42:27.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:27.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-04-25T09:42:27.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=10 2024-04-25T09:42:27.712 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:27.851 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836513 2024-04-25T09:42:27.875 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672984 2024-04-25T09:42:27.902 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509458 2024-04-25T09:42:27.913 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:30.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:30.915 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:31.078 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:42:31.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:31.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:31.079 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-04-25T09:42:31.233 INFO:tasks.workunit.client.0.smithi102.stdout:12 2024-04-25T09:42:31.233 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:31.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:31.389 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:42:31.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:31.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:31.389 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 10 2024-04-25T09:42:31.567 INFO:tasks.workunit.client.0.smithi102.stdout:10 2024-04-25T09:42:31.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:175: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:42:31.703 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:31.706 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:176: TEST_trim_max_entries_with_dups: test_log_size 1.0 9 14 2024-04-25T09:42:31.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:31.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-04-25T09:42:31.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=14 2024-04-25T09:42:31.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:31.847 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836516 2024-04-25T09:42:31.870 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672987 2024-04-25T09:42:31.894 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509461 2024-04-25T09:42:31.909 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:34.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:34.912 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:35.077 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:42:35.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:35.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:35.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-04-25T09:42:35.234 INFO:tasks.workunit.client.0.smithi102.stdout:9 2024-04-25T09:42:35.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:35.234 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:35.387 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:42:35.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:35.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:35.388 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 14 2024-04-25T09:42:35.562 INFO:tasks.workunit.client.0.smithi102.stdout:14 2024-04-25T09:42:35.562 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:177: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:42:35.682 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:35.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:178: TEST_trim_max_entries_with_dups: test_log_size 1.0 6 18 2024-04-25T09:42:35.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:35.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-04-25T09:42:35.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-04-25T09:42:35.684 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:35.829 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836519 2024-04-25T09:42:35.853 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672990 2024-04-25T09:42:35.877 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509464 2024-04-25T09:42:35.892 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:38.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:38.895 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:39.064 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:42:39.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:39.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:39.065 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-04-25T09:42:39.214 INFO:tasks.workunit.client.0.smithi102.stdout:6 2024-04-25T09:42:39.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:39.215 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:39.360 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:42:39.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:39.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:39.361 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-04-25T09:42:39.509 INFO:tasks.workunit.client.0.smithi102.stdout:18 2024-04-25T09:42:39.509 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:179: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:42:39.659 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:39.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:180: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:42:39.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:39.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:42:39.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:42:39.663 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:39.806 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836522 2024-04-25T09:42:39.829 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672993 2024-04-25T09:42:39.855 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509467 2024-04-25T09:42:39.868 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:42.870 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:42.871 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:43.069 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:42:43.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:43.069 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:43.070 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:42:43.225 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:42:43.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:43.225 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:43.383 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:43.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:43.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:43.384 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:42:43.603 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:43.603 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:183: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:42:43.720 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:43.723 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:184: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-04-25T09:42:43.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:43.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:42:43.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:42:43.724 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:43.867 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836524 2024-04-25T09:42:43.891 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672996 2024-04-25T09:42:43.914 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509470 2024-04-25T09:42:43.929 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:46.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:46.932 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:47.097 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:42:47.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:47.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:47.098 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:42:47.250 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:42:47.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:47.251 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:47.406 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:47.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:47.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:47.407 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:42:47.567 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:47.567 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:185: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:42:47.673 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:47.675 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:186: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:42:47.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:47.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:42:47.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:42:47.676 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:47.819 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836527 2024-04-25T09:42:47.843 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949672998 2024-04-25T09:42:47.867 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509472 2024-04-25T09:42:47.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:50.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:50.884 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:51.061 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:42:51.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:51.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:51.062 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:42:51.213 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:42:51.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:51.214 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:51.356 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:51.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:51.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:51.357 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:42:51.514 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:51.515 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:187: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:42:51.659 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:51.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:188: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-04-25T09:42:51.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:51.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:42:51.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:42:51.662 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:51.798 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836530 2024-04-25T09:42:51.822 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949673001 2024-04-25T09:42:51.846 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509475 2024-04-25T09:42:51.861 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:54.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:54.863 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:55.026 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:42:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:42:55.027 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:55.391 INFO:tasks.workunit.client.0.smithi102.stdout:4 2024-04-25T09:42:55.391 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:55.392 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:55.550 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:55.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:55.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:55.551 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:42:55.707 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:55.707 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:189: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:42:55.878 INFO:tasks.workunit.client.0.smithi102.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:42:55.881 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:190: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:42:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:42:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:42:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:42:55.882 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:42:56.023 INFO:tasks.workunit.client.0.smithi102.stdout:osd.0: 21474836533 2024-04-25T09:42:56.048 INFO:tasks.workunit.client.0.smithi102.stdout:osd.1: 42949673004 2024-04-25T09:42:56.072 INFO:tasks.workunit.client.0.smithi102.stdout:osd.2: 64424509478 2024-04-25T09:42:56.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:42:59.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:59.089 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:59.255 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:42:59.255 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:59.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:42:59.256 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:42:59.414 INFO:tasks.workunit.client.0.smithi102.stdout:3 2024-04-25T09:42:59.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:59.414 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:59.578 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:59.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:42:59.578 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:42:59.579 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:42:59.728 INFO:tasks.workunit.client.0.smithi102.stdout:20 2024-04-25T09:42:59.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:42:59.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:42:59.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:42:59.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:42:59.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:42:59.729 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:42:59.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:42:59.729 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:42:59.730 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:43:00.039 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:43:00.040 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:43:00.041 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:43:00.042 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:43:00.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:43:00.044 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:43:00.045 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:43:00.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:43:00.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:43:00.046 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:43:00.047 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:43:00.048 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:43:00.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:43:00.049 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:43:00.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:43:00.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:43:00.074 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:43:00.074 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493206 2024-04-25T09:43:00.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:43:00.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:43:00.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:43:00.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/repro-long-log 0 2024-04-25T09:43:00.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:43:00.076 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:43:00.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:43:00.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:43:00.077 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:43:00.077 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:43:00.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:43:00.078 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:43:00.081 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:43:00.081 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:43:00.082 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:43:00.083 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:43:00.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:43:00.084 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:43:00.084 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:43:00.085 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:43:00.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:43:00.086 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:43:00.086 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:43:00.087 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:43:00.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:43:00.088 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:43:00.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:43:00.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:43:00.090 INFO:tasks.workunit.client.0.smithi102.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.493206 2024-04-25T09:43:00.090 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.493206 2024-04-25T09:43:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:43:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:43:00.091 INFO:tasks.workunit.client.0.smithi102.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:43:00.093 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:43:00.093 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:43:00.105 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-04-25T09:43:00.105 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T09:43:00.465 DEBUG:teuthology.parallel:result is None 2024-04-25T09:43:00.466 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:43:00.477 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:43:00.478 DEBUG:teuthology.orchestra.run.smithi102:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T09:43:00.534 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:43:00.534 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T09:43:00.544 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T09:43:00.544 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T09:43:00.613 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T09:43:00.622 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T09:43:00.622 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T09:43:00.648 INFO:teuthology.orchestra.run.smithi102.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T09:43:00.648 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================== 2024-04-25T09:43:00.648 INFO:teuthology.orchestra.run.smithi102.stdout:+hv01.front.sepi 67.205.162.81 3 u 209 256 377 0.164 -3.820 0.398 2024-04-25T09:43:00.648 INFO:teuthology.orchestra.run.smithi102.stdout:+hv02.front.sepi 63.231.80.2 3 u 66 256 377 0.171 -0.551 0.552 2024-04-25T09:43:00.648 INFO:teuthology.orchestra.run.smithi102.stdout:*hv03.front.sepi 74.6.168.72 3 u 71 256 377 0.140 -2.797 0.407 2024-04-25T09:43:00.648 INFO:teuthology.orchestra.run.smithi102.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-25T09:43:00.648 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T09:43:00.656 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T09:43:00.657 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T09:43:00.686 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T09:43:00.719 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T09:43:00.752 INFO:teuthology.task.internal:Duration was 10033.045265 seconds 2024-04-25T09:43:00.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T09:43:00.786 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T09:43:00.786 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T09:43:00.831 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T09:43:00.832 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-04-25T09:43:00.832 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T09:43:00.885 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T09:43:00.885 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T09:43:00.966 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T09:43:00.967 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T09:43:01.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T09:43:01.070 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T09:43:01.070 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T09:43:01.083 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T09:43:01.095 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:43:01.134 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-04-25T09:43:01.146 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:43:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:43:01.192 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T09:43:01.201 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T09:43:01.203 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672923/remote/smithi102 2024-04-25T09:43:01.204 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T09:43:01.258 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T09:43:01.259 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T09:43:01.300 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T09:43:01.311 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T09:43:01.311 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T09:43:01.337 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T09:43:01.337 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T09:43:01.348 INFO:teuthology.orchestra.run.smithi102.stdout: 658467 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 09:43 /home/ubuntu/cephtest 2024-04-25T09:43:01.349 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T09:43:01.371 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T09:43:01.439 INFO:teuthology.run:Summary data: description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} duration: 10033.045264720917 flavor: default owner: scheduled_nmordech@teuthology success: true 2024-04-25T09:43:01.440 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T09:43:01.521 INFO:teuthology.run:pass